Work_Queue::new ( )
Work_Queue::new ( $port )
Work_Queue::new ( port => ..., name => ..., catalog => ..., shutdown => ...)
name
port
stats
stats_hierarchy
stats_category
enable_monitoring($dir_name)
enable_monitoring_full($dir_name)
activate_fast_abort
activate_fast_abort_category
empty
hungry
specify_algorithm
specify_task_order
specify_name
specify_priority
specify_num_tasks_left
specify_master_mode
specify_catalog_server
specify_log
specify_password
specify_password_file
cancel_by_taskid
cancel_by_tasktag
shutdown_workers
blacklist
blacklist_clear
specify_keepalive_interval
specify_keepalive_timeout
estimate_capacity
activate_worker_waiting
tune
specify_max_resources
specify_max_category_resources
initialize_categories
submit
wait
Work_Queue - Perl Work Queue bindings.
The objects and methods provided by this package correspond to the native C API in work_queue.h. See also Work_Queue::Task, which is automatically loaded with this module.
The SWIG-based Perl bindings provide a higher-level interface, such as:
use Work_Queue;
my $q = Work_Queue->new( port => $port, name => 'my_queue_name');
my $t = Work_Queue::Task->new($command); $t->specify_input_file(local_name => 'some_name', remote_name => 'some_other_name'); $t->specify_output_file('some_name');
$q->submit($t);
my $stats = $q->stats; print $stats->{tasks_running}, '\n';
$t = $q->wait(5);
if($t) { my $resources = $t->resources_measured; print $resources->{resident_memory}, '\n'; }
Work_Queue::new ( )
Work_Queue::new ( $port )
Work_Queue::new ( port => ..., name => ..., catalog => ..., shutdown => ...)
Create a new work queue.
The port number to listen on. If zero is specified, then the default is chosen, and if -1 is specified, a random port is chosen.
The project name to use.
Whether or not to enable catalog mode.
Automatically shutdown workers when queue is finished. Disabled by default.
my $q = Work_Queue->new( port => 0, name => 'my_queue' );
See work_queue_create in the C API for more information about environmental variables that affect the behavior this method.
name
Get the project name of the queue.
print $q->name;
port
Get the listening port of the queue.
print $q->port
stats
Get the master's queue statistics.
print $q->stats->{workers_busy};
stats_hierarchy
Get the queue statistics, including master and foremen.
print $q->stats_hierarchy->{workers_busy};
stats_category
Get the tasks statistics from the particular category.
$s = $q->stats_category("my_category") print $s->{tasks_waiting}
enable_monitoring($dir_name)
Enables resource monitoring of tasks in the queue, and writes a summary per task to the directory given. Additionally, all summaries are consolidate into the file all_summaries-PID.log
Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).
enable_monitoring_full($dir_name)
As @ref enable_monitoring, but it also generates a time series and a debug file. WARNING: Such files may reach gigabyte sizes for long running tasks.
Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).
activate_fast_abort
Turn on or off fast abort functionality for a given queue for tasks without an explicit category. Given the multiplier, abort a task which running time is larger than the average times the multiplier. Fast-abort is computed per task category. The value specified here applies to all the categories for which @ref activate_fast_abort_category was not explicitely called.
The multiplier of the average task time at which point to abort; if less than zero, fast_abort is deactivated (the default).
activate_fast_abort_category
Turn on or off fast abort functionality for a given category. Given the multiplier, abort a task which running time is larger than the average times the multiplier. The value specified here applies only to tasks in the given category. (Note: work_queue_activate_fast_abort_category(q, ``default'', n) is the same as work_queue_activate_fast_abort(q, n).)
The name of the category.
The multiplier of the average task time at which point to abort; if zero, fast_abort is deactivated. If less than zero (default), use the fast abort of the ``default'' category.
empty
Determine whether there are any known tasks queued, running, or waiting to be collected.
Returns 0 if there are tasks remaining in the system, 1 if the system is ``empty''.
hungry
Determine whether the queue can support more tasks.
Returns the number of additional tasks it can support if ``hungry'' and 0 if ``sated''.
specify_algorithm
Set the worker selection algorithm for queue.
One of the following algorithms to use in assigning a task to a worker:
specify_task_order
Set the order for dispatching submitted tasks in the queue to workers:
One of the following algorithms to use in dispatching
specify_name
Change the project name for the given queue.
The new project name.
specify_priority
Change the project priority for the given queue.
An integer that presents the priorty of this work queue master. The higher the value, the higher the priority.
specify_num_tasks_left
Specify the number of tasks not yet submitted to the queue. It is used by work_queue_factory to determine the number of workers to launch. If not specified, it defaults to 0. work_queue_factory considers the number of tasks as: num tasks left + num tasks running + num tasks read.
ntasks Number of tasks yet to be submitted.
specify_master_mode
Specify the master mode for the given queue.
This may be one of the following values:
specify_catalog_server
Specify the catalog server the master should report to.
The hostname of the catalog server.
The port the catalog server is listening on.
specify_log
Specify a log file that records the states of connected workers and submitted tasks.
Name of the file to write the log. If the file exists, then new records are appended.
specify_password
Add a mandatory password that each worker must present.
The password, as a string.
specify_password_file
Add a mandatory password file that each worker must present.
Name of the file containing the password.
cancel_by_taskid
Cancel task identified by its taskid and remove from the given queue.
The taskid returned from Work_Queue->submit.
cancel_by_tasktag
Cancel task identified by its tag and remove from the given queue.
The tag assigned to task using $t->speficy_tag($tag);
shutdown_workers
Shutdown workers connected to queue. Gives a best effort and then returns the number of workers given the shutdown order.
The number to shutdown. To shut down all workers, specify 0.
blacklist
Blacklist workers running on host.
The hostname the host running the workers.
blacklist_clear
Remove host from blacklist. Clear all blacklist if host not provided.
The of the hostname the host.
specify_keepalive_interval
Change keepalive interval for a given queue.
specify_keepalive_timeout
Change keepalive timeout for a given queue.
Minimum number of seconds to wait for a keepalive response from worker before marking it as dead.
estimate_capacity
Turn on master capacity measurements.
activate_worker_waiting
Wait for at least n workers to connect before continuing.
Number of workers.
tune
Tune advanced parameters for work queue. Return 0 on succes, -1 on failure.
Treat each worker as having (actual_cores * multiplier) total cores. (default = 1.0)
Treat each worker as having an additional ``modifier'' cores. (default=0)
Set the minimum number of seconds to wait for files to be transferred to or from a worker. (default=300)
Set the minimum number of seconds to wait for files to be transferred to or from a foreman. (default=3600)
Set the multiplier of the average task time at which point to abort; if negative or zero fast_abort is deactivated. (default=0)
Set the minimum number of seconds to wait before sending new keepalive checks to workers. (default=300)
Set the minimum number of seconds to wait for a keepalive response from worker before marking it as dead. (default=30)
specify_max_resources
Enables resource autolabeling for tasks without an explicit category (``default''
category). rm specifies the maximum resources a task in the default category
may use. If rm is undefined
, disable autolabeling for the default category.
Hash reference indicating maximum values. See @resources_measured for possible fields.
A maximum of 4 cores is found on any worker:
q->specify_max_resources({'cores' => 4});
A maximum of 8 cores, 1GB of memory, and 10GB disk are found on any worker:
q->specify_max_resources({'cores' => 8, 'memory' => 1024, 'disk' => 10240});
specify_max_category_resources
Enables resource autolabeling for tasks in the given category.
rm specifies the maximum resources a task in the category may use.
If rm is undefined
, disable autolabeling for that category.
Name of the category
Hash reference indicating maximum values. See @resources_measured for possible fields.
A maximum of 4 cores is found on any worker:
q->specify_max_category_resources('my_category', {'cores' => 4});
A maximum of 8 cores, 1GB of memory, and 10GB disk are found on any worker:
q->specify_max_category_resources('my_category', {'cores' => 8, 'memory' => 1024, 'disk' => 10240});
initialize_categories
Initialize first value of categories
Hash reference indicating maximum values. See @resources_measured for possible fields.
JSON file with resource summaries.
submit
Submit a task to the queue.
A task description created from Work_Queue::Task.
$q->submit($task);
wait
Wait for tasks to complete.
This call will block until the timeout has elapsed
The number of seconds to wait for a completed task back before returning. Use an integer to set the timeout or the constant $Work_Queue::WORK_QUEUE_WAITFORTASK to block until a task has completed.
while( !$q->empty ) { ... $task = $q->wait($seconds);
if($task) { ... } ... } =cut