cctools
|
TaskVine Manager specialized to compute dask graphs. More...
Public Member Functions | |
def | get |
Execute the task graph dsk and return the results for keys in graph. More... | |
TaskVine Manager specialized to compute dask graphs.
Managers created via DaskVine can be used to execute dask graphs via the method ndcctools.taskvine.dask_executor.DaskVine.get as follows:
Parameters for execution can be set as arguments to the compute function. These arguments are applied to each task executed:
def ndcctools.taskvine.dask_executor.DaskVine.get | ( | self, | |
dsk, | |||
keys, | |||
environment = None , |
|||
extra_files = None , |
|||
lazy_transfers = False , |
|||
low_memory_mode = False , |
|||
checkpoint_fn = None , |
|||
resources = None , |
|||
resources_mode = 'fixed' , |
|||
retries = 5 , |
|||
verbose = False |
|||
) |
Execute the task graph dsk and return the results for keys in graph.
dsk | The task graph to execute. |
keys | A single key or a possible nested list of keys to compute the value from dsk. |
environment | A taskvine file representing an environment to run the tasks. |
extra_files | A dictionary of {taskvine.File: "remote_name"} to add to each task. |
lazy_transfers | Whether to keep intermediate results only at workers (True) or to bring back each result to the manager (False, default). True is more IO efficient, but runs the risk of needing to recompute results if workers are lost. |
low_memory_mode | Split graph vertices to reduce memory needed per function call. It removes some of the dask graph optimizations, thus proceed with care. |
checkpoint_fn | When using lazy_transfers, a predicate with arguments (dag, key) called before submitting a task. If True, the result is brought back to the manager. |
resources | A dictionary with optional keys of cores, memory and disk (MB) to set maximum resource usage per task. |
resources_mode | Automatically resize allocation per task. One of 'fixed' (use the value of 'resources' above), 'max througput', 'max' (for maximum values seen), 'min_waste', 'greedy bucketing' or 'exhaustive bucketing'. This is done per function type in dsk. |
retries | Number of times to attempt a task. Default is 5. |
verbose | if true, emit additional debugging information. |