Public Member Functions
work_queue.Factory Class Reference

Launch a Work Queue factory. More...

Inheritance diagram for work_queue.Factory:

Public Member Functions

def __init__
 Create a factory for the given batch_type and manager name. More...
def start
 Start a factory process. More...
def stop
 Stop the factory process. More...

Detailed Description

Launch a Work Queue factory.

The command line arguments for work_queue_factory can be set for a factory object (with dashes replaced with underscores). Creating a factory object does not immediately launch it, so this is a good time to configure the resources, number of workers, etc. Factory objects function as Python context managers, so to indicate that a set of commands should be run with a factory running, wrap them in a with statement. The factory will be cleaned up automatically at the end of the block. You can also make config changes to the factory while it is running. As an example,

# normal WQ setup stuff
workers = work_queue.Factory("sge", "myproject")
workers.cores = 4
with workers:
    # submit some tasks
    workers.max_workers = 300
    # got a pile of tasks, allow more workers
# any additional cleanup steps on the manager 

Constructor & Destructor Documentation

def work_queue.Factory.__init__ (   self,
  manager_name = None,
  manager_host_port = None,
  factory_binary = None,
  worker_binary = None,
  log_file = os.devnull 

Create a factory for the given batch_type and manager name.

manager_name or, manager_host_port should be specified. If factory_binary or worker_binary is not specified, $PATH will be searched.

References work_queue.Factory._config_file, work_queue.Factory._factory_binary, work_queue.Factory._factory_proc, work_queue.Factory._find_exe(), work_queue.Factory._log_file, work_queue.Factory._opts, work_queue.Factory._set_manager(), and work_queue.Factory._write_config().

Member Function Documentation

def work_queue.Factory.start (   self)

Start a factory process.

It's best to use a context manager (with statement) to automatically handle factory startup and tear-down. If another mechanism will ensure cleanup (e.g. running inside a container), manually starting the factory may be useful to provision workers from inside a Jupyter notebook.

References work_queue.Factory._config_file, work_queue.Factory._construct_command_line(), work_queue.Factory._factory_proc, work_queue.Factory._log_file, work_queue.Factory._write_config(), and work_queue.Factory.scratch_dir.

Referenced by work_queue.Factory.stop().

def work_queue.Factory.stop (   self)

Stop the factory process.

References work_queue.Factory._config_file, work_queue.PythonTask._env_file, work_queue.Factory._env_file, work_queue.Factory._factory_proc, work_queue.Factory._opts, rmsummary.snapshots_count, rmsummary.start, rmonitor_bw_info.start, work_queue.Factory.start(), and work_queue.Factory.stop().

Referenced by work_queue.Factory.stop().

The documentation for this class was generated from the following file: