qpym2.run.create_staging#

Create the staging files.

TODO:

  • Check segfault errors while running multiprocessing, seem to happen with generic errors.

  • Refactor individual calls to be made from multiprocessing code?

Module Contents#

Functions#

create_staging

Create the staging files.

_create_rdf_partial

Create a partial function for create_rdf_m2mc to be run with multiprocessing.

create_staging_data

Create the staging files.

create_staging_mp

Create the staging files.

create_staging_all_mp

Create MC, 0v and data staging files in parallel.

create_staging_0v

Create the staging files.

Data#

_NDBD_STAGING_NAME

API#

qpym2.run.create_staging._NDBD_STAGING_NAME = 'ndbd'#
qpym2.run.create_staging.create_staging(cfg)#

Create the staging files.

All the configuration is in the cfg object.

qpym2.run.create_staging._create_rdf_partial(**kwargs)#

Create a partial function for create_rdf_m2mc to be run with multiprocessing.

TODO: Use mp.Value and mp.Array to share common varaibles between processes.

qpym2.run.create_staging.create_staging_data(cfg)#

Create the staging files.

Args:

cfg (dict): configuration dictionary that holds all the configuration settings.

TODO: not tested.

qpym2.run.create_staging.create_staging_mp(cfg, multiproc=True, ncores=None)#

Create the staging files.

Args:

cfg (dict): configuration dictionary that holds all the configuration settings. multiproc (bool): if True, use multiprocessing. ncores (int): number of cores to use. If None, use all available cores.

qpym2.run.create_staging.create_staging_all_mp(cfg, multiproc=True, ncores=None)#

Create MC, 0v and data staging files in parallel.

Args:

cfg (dict): configuration dictionary that holds all the configuration settings. multiproc (bool): if True, use multiprocessing. ncores (int): number of cores to use. If None, use all available cores.

TODO: this actually does not run in parallel, we need to replace map method with

an async method and read the output later.

qpym2.run.create_staging.create_staging_0v(cfg)#

Create the staging files.

Args:

cfg (dict): configuration dictionary that holds all the configuration settings.