Entry Point

APIs to manage backends and integrate the toolkit with other projects.

parfun.entry_point.add_parallel_options(parser: ArgumentParser) → None

Adds argparse options required to initialize this parallel toolkit.

Return type:

None

parfun.entry_point.get_parallel_backend() → BackendEngine | None

Returns:

the current backend instance, or None if no backend is currently set.

Return type:

Optional[BackendEngine]

parfun.entry_point.set_parallel_backend(backend: str | BackendEngine, *args, **kwargs) → None

Initializes and sets the current parfun backend.

set_parallel_backend("local_multiprocessing", max_workers=4, is_process=False)

Parameters:

• backend (Union[str, BackendEngine]) –

  Supported backend options:

  • "none": disable the current parallel backend.

    Functions decorated with parfun() will run sequentially as if not decorated.

    Partitionning and combining functions will be ignored.

  • "local_single_process": runs the tasks inside the calling Python process.

    Functions decorated with parfun() will partition the input data, and run the combining function on the output data, but will also execute the function inside the calling Python process.

    Mostly intended for debugging purposes.

    See LocalSingleProcessBackend.

  • "local_multiprocessing": runs the tasks in parallel using Python multiprocessing processes.

    See LocalMultiprocessingBackend.

  • "scaler_local": runs the tasks in parallel using an internally managed Scaler cluster.

    See ScalerLocalBackend.

  • "scaler_remote": runs the tasks in parallel using an externally managed Dask cluster.

    See ScalerRemoteBackend.

  • "dask_local": runs the tasks in parallel using an internally managed Dask cluster.

    See DaskLocalClusterBackend.

  • "dask_remote": runs the tasks in parallel using an externally managed Dask cluster.

    See DaskRemoteClusterBackend.

  • "dask_current": runs the tasks in parallel using the currently running Dask client (get_client()).

    See DaskCurrentBackend.

• args – Additional positional parameters for the backend constructor

• kwargs – Additional keyword parameters for the backend constructor.

Return type:

None

parfun.entry_point.set_parallel_backend_context(backend: str | BackendEngine, *args, **kwargs)

Sets a new parallel backend instance in a contextlib’s context.

with set_parallel_backend_context("local_single_processing"):
    some_parallel_computation()

Parameters:

backend (Union[str, BackendEngine]) – See set_parallel_backend().