cctools
ndcctools.taskvine.task.PythonTask Class Reference
Inheritance diagram for ndcctools.taskvine.task.PythonTask:
ndcctools.taskvine.task.Task

Public Member Functions

def __init__
 Creates a new python task. More...
 
def submit_finalize
 Finalizes the task definition once the manager that will execute is run. More...
 
def enable_temp_output
 Marks the output of this task to stay at the worker. More...
 
def set_output_cache
 Set the cache behavior for the output of the task. More...
 
def output_file
 Returns the ndcctools.taskvine.file.File object that represents the output of this task. More...
 
def output
 returns the result of a python task as a python variable More...
 
def disable_output_serialization
 Disables serialization of results to disk when writing to a file for transmission. More...
 
- Public Member Functions inherited from ndcctools.taskvine.task.Task
def __init__
 Create a new task specification. More...
 
def submit_finalize
 Finalizes the task definition once the manager that will execute is run. More...
 
def clone
 Return a copy of this task. More...
 
def set_command
 Set the command to be executed by the task. More...
 
def set_coprocess
 Set the coprocess at the worker that should execute the task's command. More...
 
def set_scheduler
 Set the worker selection scheduler for task. More...
 
def set_tag
 Attach a user defined logical name to the task. More...
 
def set_category
 Label the task with the given category. More...
 
def add_feature
 Label the task with the given user-defined feature. More...
 
def add_input
 Add any input object to a task. More...
 
def add_output
 Add any output object to a task. More...
 
def set_snapshot_file
 When monitoring, indicates a json-encoded file that instructs the monitor to take a snapshot of the task resources. More...
 
def add_environment
 Adds an execution environment to the task. More...
 
def set_retries
 Indicate the number of times the task should be retried. More...
 
def set_cores
 Indicate the number of cores required by this task. More...
 
def set_memory
 Indicate the memory (in MB) required by this task. More...
 
def set_disk
 Indicate the disk space (in MB) required by this task. More...
 
def set_gpus
 Indicate the number of GPUs required by this task. More...
 
def set_priority
 Indicate the the priority of this task (larger means better priority, default is 0). More...
 
def set_env_var
 Set this environment variable before running the task. More...
 
def set_monitor_output
 Set a name for the resource summary output directory from the monitor. More...
 
def tag
 Get the user-defined logical name for the task. More...
 
def category
 Get the category name for the task. More...
 
def command
 Get the shell command executed by the task. More...
 
def std_output
 Get the standard output of the task. More...
 
def output
 Get the standard output of the task. More...
 
def id
 Get the task id number. More...
 
def exit_code
 Get the exit code of the command executed by the task. More...
 
def result
 Return a string that explains the result of a task. More...
 
def completed
 Return True if task executed and its command terminated normally. More...
 
def successful
 Return True if task executed successfully, (i.e. More...
 
def get_metric
 Return various integer performance metrics about a completed task. More...
 
def addrport
 Get the address and port of the host on which the task ran. More...
 
def hostname
 Get the address and port of the host on which the task ran. More...
 
def resources_measured
 Get the resources measured for the task execution if resource monitoring is enabled. More...
 
def limits_exceeded
 Get the resources the task exceeded. More...
 
def resources_requested
 Get the resources the task requested to run. More...
 
def resources_allocated
 Get the resources allocated to the task in its latest attempt. More...
 

Constructor & Destructor Documentation

def ndcctools.taskvine.task.PythonTask.__init__ (   self,
  func,
  args,
  kwargs 
)

Creates a new python task.

Parameters
selfReference to the current python task object
funcpython function to be executed by task
argsarguments used in function to be executed by task
kwargskeyword arguments used in function to be executed by task

Member Function Documentation

def ndcctools.taskvine.task.PythonTask.submit_finalize (   self,
  manager 
)

Finalizes the task definition once the manager that will execute is run.

This function is run by the manager before registering the task for execution.

Parameters
selfReference to the current python task object
managerManager to which the task was submitted
def ndcctools.taskvine.task.PythonTask.enable_temp_output (   self)

Marks the output of this task to stay at the worker.

Functions that consume the output of this tasks have to add self.output_file as an input, and cloudpickle.load() it.

E.g.:

1 ta = PythonTask(fn, ...)
2 ta.enable_temp_output()
3 tid = m.submit(ta)
4 
5 t = m.wait(...)
6 if t.id == tid:
7  tb = PythonTask(fn_with_tmp, "ta_output.file")
8  tb.add_input(ta.output_file, "ta_output.file")
9  m.submit(tb)

where fn_with_tmp may look something like this:

1 def fn_with_tmp(filename):
2  import cloudpickle
3  with open(filename) as f:
4  data = cloudpickle.load(f)
Parameters
selfReference to the current python task object
def ndcctools.taskvine.task.PythonTask.set_output_cache (   self,
  cache = False 
)

Set the cache behavior for the output of the task.

Parameters
cacheIf True or 'workflow', cache the file at workers for reuse until the end of the workflow. If 'always', the file is cache until the end-of-life of the worker. Default is False (file is not cache).
def ndcctools.taskvine.task.PythonTask.output_file (   self)

Returns the ndcctools.taskvine.file.File object that represents the output of this task.

def ndcctools.taskvine.task.PythonTask.output (   self)

returns the result of a python task as a python variable

Parameters
selfreference to the current python task object
def ndcctools.taskvine.task.PythonTask.disable_output_serialization (   self)

Disables serialization of results to disk when writing to a file for transmission.

WARNING: Only do this if the function itself encodes the output in a way amenable for serialization.

Parameters
selfReference to the current python task object

The documentation for this class was generated from the following file: