Skip to content

Commit

Permalink
Add reduce method to allow for pickling of Parallel table objects.
Browse files Browse the repository at this point in the history
  • Loading branch information
JSKenyon committed Feb 23, 2022
1 parent cdd2d3f commit 5cad3da
Showing 1 changed file with 12 additions and 0 deletions.
12 changes: 12 additions & 0 deletions daskms/parallel_table.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@
log = logging.getLogger(__name__)


def _map_create_parallel_table(cls, args, kwargs):
""" Support pickling of kwargs in ParallelTable.__reduce__ """
return cls(*args, **kwargs)


def _parallel_table_finalizer(_linked_tables):

for table in _linked_tables.values():
Expand All @@ -31,6 +36,13 @@ def __init__(self, *args, **kwargs):

finalize(self, _parallel_table_finalizer, self._linked_tables)

def __reduce__(self):
""" Defer to _map_create_parallel_table to support kwarg pickling """
return (
_map_create_parallel_table,
(ParallelTable, self._args, self._kwargs)
)

def getcol(self, *args, **kwargs):
table = self._get_table(threading.get_ident())
return table.getcol(*args, **kwargs)
Expand Down

3 comments on commit 5cad3da

@sjperkins
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW you can make the factory function a classmethod. Nice to keep things in one place

@JSKenyon
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! The problem now is how to get this to behave like the TableProxy in dask-ms. Unfortunately I cannot figure out how to use the metaclass trick as the Table class is a Boost.Python.class i.e. one gets TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases and I cannot think of a simple way to resolve this. I thought that maybe it would be possible to do it using the __new__ method to check the cache on instantiation but I cannot quite get that right. Oh well, learning loads.

@sjperkins
Copy link
Member

@sjperkins sjperkins commented on 5cad3da Feb 23, 2022 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.