DAOS-16355 client: pydaos.torch module (#15475) #15536
Conversation
|
Ticket title is 'pydaos.torch modules' |
mjmac
changed the title
|
Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/379/log |
|
Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/387/log |
|
Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/385/log |
|
Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/295/log |
|
Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/273/log |
|
Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/317/log |
This patch marks all pool and container handles as if they were created with g2l in the child processes after fork. It prevents misinteractions if one of the child processes closes the handle. The marking is done by iterating through all the pool and container handles which was not supported by the hhash code. This patch also: - adds support for fork to pydaos. - introduces daos_reinit() to be called after fork. - fixes IL to set the atfork callback when no extra eq are used. - remove support for creating an event queue for each pydaos put/get operation. This makes the global event queue the only option. This should probably be moved to a per-thread eq in the future. Signed-off-by: Johann Lombardi <[email protected]>
Disable call to pthread_atfork and daos_reinit() in pydaos until DAOS-16637 is understood. Signed-off-by: Johann Lombardi <[email protected]>
Introducing pydaos.torch module that allows use DAOS POSIX containers as a datasource for pytorch framework in form of pydaos.torch.Dataset and pydaos.torch.IterableDataset classes. Signed-off-by: Denis Barakhtanov <[email protected]>
mjmac
force-pushed
the
mjmac/DAOS-16355-google-2.6
branch
from
117c94f
to
e50b3c3
Compare
Introducing pydaos.torch module that allows use DAOS POSIX containers
as a datasource for pytorch framework in form of pydaos.torch.Dataset and
pydaos.torch.IterableDataset classes.
Signed-off-by: Denis Barakhtanov [email protected]