Backend

class parfun.backend.mixins.BackendEngine

Asynchronous task manager interface.

abstract allows_nested_tasks() → bool

Indicates if Parfun can submit new tasks from other tasks.

abstract session() → BackendSession

Returns a new managed session for submitting tasks.

with backend.session() as session:
    arg_ref = session.preload_value(arg)

    future = session.submit(fn, arg_ref)

    print(future.result())

abstract shutdown()

Shutdowns all resources required by the backend engine.

class parfun.backend.mixins.BackendSession

A task submitting session to a backend engine that manages the lifecycle of the task objects (preloaded values, argument values and future objects).

preload_value(value: Any) → Any

Preloads a value to the backend engine.

The returned value will be used when calling submit() instead of the original value.

abstract submit(fn: Callable, *args, **kwargs) → ProfiledFuture

Executes an asynchronous computation.

Blocking if no computing resource is available.

class parfun.backend.local_single_process.LocalSingleProcessBackend

allows_nested_tasks() → bool

Indicates if Parfun can submit new tasks from other tasks.

session() → BackendSession

Returns a new managed session for submitting tasks.

with backend.session() as session:
    arg_ref = session.preload_value(arg)

    future = session.submit(fn, arg_ref)

    print(future.result())

shutdown()

Shutdowns all resources required by the backend engine.

class parfun.backend.local_single_process.LocalSingleProcessSession

submit(fn: Callable, *args, **kwargs) → ProfiledFuture

Executes an asynchronous computation.

Blocking if no computing resource is available.

class parfun.backend.local_multiprocessing.LocalMultiprocessingBackend(max_workers: int = 1, is_process: bool = True, **kwargs)

Concurrent engine that uses Python builtin multiprocessing module.

allows_nested_tasks() → bool

Indicates if Parfun can submit new tasks from other tasks.

session() → LocalMultiprocessingSession

Returns a new managed session for submitting tasks.

with backend.session() as session:
    arg_ref = session.preload_value(arg)

    future = session.submit(fn, arg_ref)

    print(future.result())

shutdown(wait=True)

Shutdowns all resources required by the backend engine.

class parfun.backend.local_multiprocessing.LocalMultiprocessingSession(underlying_executor: Executor)

submit(fn, *args, **kwargs) → ProfiledFuture

Executes an asynchronous computation.

Blocking if no computing resource is available.

class parfun.backend.dask.DaskBaseBackend(n_workers: int)

allows_nested_tasks() → bool

Indicates if Parfun can submit new tasks from other tasks.

session() → DaskSession

Returns a new managed session for submitting tasks.

with backend.session() as session:
    arg_ref = session.preload_value(arg)

    future = session.submit(fn, arg_ref)

    print(future.result())

class parfun.backend.dask.DaskCurrentBackend(n_workers: int)

Uses the current Dask worker context to deduce the backend instance.

This backend should be used by Dask’s worker tasks that desire to access the underlying backend instance.

shutdown()

Shutdowns all resources required by the backend engine.

class parfun.backend.dask.DaskLocalClusterBackend(n_workers: int = 1, dashboard_address=':33333', memory_limit='100GB')

Creates a Dask cluster on the local machine and uses it as a backend engine.

shutdown()

Shutdowns all resources required by the backend engine.

class parfun.backend.dask.DaskRemoteClusterBackend(scheduler_address: str)

Connects to a previously instantiated Dask instance as a backend engine.

shutdown()

Shutdowns all resources required by the backend engine.

class parfun.backend.dask.DaskSession(engine: DaskBaseBackend, n_workers: int)

submit(fn, *args, **kwargs) → ProfiledFuture | None

Executes an asynchronous computation.

Blocking if no computing resource is available.

class parfun.backend.scaler.ScalerLocalBackend(per_worker_queue_size: int, scheduler_address: str | None = None, n_workers: int = 1, allows_nested_tasks: bool = True, **kwargs)

Creates a Scaler cluster on the local machine and uses it as a backend engine.

shutdown()

Shutdowns all resources required by the backend engine.

class parfun.backend.scaler.ScalerRemoteBackend(scheduler_address: str, n_workers: int = 1, allows_nested_tasks: bool = True, **client_kwargs)

Connects to a previously instantiated Scaler instance as a backend engine.

allows_nested_tasks() → bool

Indicates if Parfun can submit new tasks from other tasks.

session() → ScalerSession

Returns a new managed session for submitting tasks.

with backend.session() as session:
    arg_ref = session.preload_value(arg)

    future = session.submit(fn, arg_ref)

    print(future.result())

shutdown()

Shutdowns all resources required by the backend engine.

class parfun.backend.scaler.ScalerSession(scheduler_address: str, n_workers: int, **kwargs)

preload_value(value: Any) → ObjectReference

Preloads a value to the backend engine.

The returned value will be used when calling submit() instead of the original value.

submit(fn, *args, **kwargs) → ProfiledFuture | None

Executes an asynchronous computation.

Blocking if no computing resource is available.