API

parfun.parfun(split: ~typing.Callable[[~parfun.kernel.function_signature.NamedArguments], ~typing.Tuple[~parfun.kernel.function_signature.NamedArguments, ~typing.Generator[~parfun.kernel.function_signature.NamedArguments, None, None] | ~typing.Generator[~typing.Tuple[int, ~parfun.kernel.function_signature.NamedArguments] | None, int, None]]], combine_with: ~typing.Callable[[~typing.Iterable[~typing.Any]], ~typing.Any], initial_partition_size: int | ~typing.Callable[[~typing.Any], int] | None = None, fixed_partition_size: int | ~typing.Callable[[~typing.Any], int] | None = None, profile: bool = False, trace_export: str | None = None, partition_size_estimator_factory: ~typing.Callable[[], ~parfun.partition_size_estimator.mixins.PartitionSizeEstimator] = <class 'parfun.partition_size_estimator.linear_regression_estimator.LinearRegessionEstimator'>) → Callable

Returns a function decorator that automatically parallelizes a function.

@parfun(
    split=per_argument(
        values=lists_by_chunk,
    ),
    combine_with=lists_concat
)
def multiply_by_constant(values: Iterable[int], constant: int):
    return [v * constant for v in values]

# This would be functionally equivalent to running the function inside a single for loop:

results = []
for partition in lists_by_chunk(values):
    results.append(multiply_by_constant(partition, constant))

return combine_with(results)
Parameters:

  • split –

    Partition the data based on the provided partitioning function.

    See api for the list of predefined partitioning functions.

  • combine_with (Callable) – aggregates the results by running the function.

  • initial_partition_size (int | Callable[[PartitionType], int] | None) –

    Overrides the first estimate from the partition size estimator.

    If the value is a callable, the function will be provided with the input to be partitioned and shall return the initial partition size to use.

  • fixed_partition_size (int | Callable[[PartitionType], int] | None) –

    Uses a constant partition size and do not run the partition size estimator.

    If the value is a callable, the function will be provided with the input to be partitioned and shall return the partition size to use.

  • profile (bool) – if true, prints additional debugging information about the parallelization overhead.

  • trace_export (str) – if defined, will export the execution time to the provided CSV file’s path.

  • partition_size_estimator_factory (Callable[[], PartitionSizeEstimator]) – the partition size estimator class to use

Returns:

a decorated function

Return type:

Callable