Practice Problems for Work Queue

Problem 1: Write a Work Queue program to compute the maxima of a simulator.
Problem 2: Run Work Queue on multiple platforms
1. Objectives
2. Problem Statement
Problem 3: Apply Work Queue in your research
1. Problem Statement

Use the information in the Work Queue page to learn more about Work Queue, if necessary.

Problem 1

Write a Work Queue program to compute the maxima of a simulator.

Objectives

This problem will illustrate the following:

Work Queue's support for the construction of large dynamic workflows.
The execution model of Work Queue.

Problem Statement

This problem involves a simulator that takes an integer as input (x), applies this input in a function (y=f(x)), and returns the evaluated value (y) as output.

The simulator for this problem is a Python executable. It can be downloaded using:

> wget http://www.nd.edu/~ccl/software/tutorials/ndtut12/wq/simulator.py

It can be executed by specifying an integer as a command-line input argument. For example, to run the simulator with an input of 10 and write the output value to a file named 10.output:

> python simulator.py 10 > 10.output

Write a Work Queue program that determines the maxima of this simulator. You will achieve this without any knowledge of the function (f(x)) implemented in the simulator. That is, you will only use the output values returned by the simulator for the given inputs to determine its maxima. Or in other words, you will treat the simulator as a black box.

Hints

Determine the search space (values of x) that you want to explore (e.g., from -300 to 300). Also decide on the granularity at which you want to explore the search space (e.g., in increments of 20).
Select values from the search space with the chosen granularity to feed as inputs to the simulator. Create a Work Queue task to run the simulator with a given input value.
You can apply techniques similar to binary search to narrow down the search space and lower your search granularity as you gather outputs for the input values fed to the simulator.
When you have narrowed down your search space to a single value, you have found the maxima. Note that this maxima is valid only for the chosen search space. Therefore, to determine the global maxima, make sure to chose your search space and search granularity carefully.

For the given simulator, the value of the (global) maxima can be found here.

Problem 2

Configure Work Queue to run on multiple platforms

Objectives

This problem will familiarize participants with the following:

Setup Work Queue programs to harness resources from multiple platforms.

Problem Statement

For this problem, you will run the Work Queue program you developed for Problem 1 on multiple platforms. To run the tasks in your Work Queue program, you will start multiple workers on these platforms: ND CRC and personal machines such as desktops or laptops.

In this assignment, you will use the catalog server and project name feature to have the workers automatically find and establish connection with the master without being provided with the master's hostname and port.

To do this, you will first modify your Work Queue master program to use the default catalog server (running at chirp.cse.nd.edu) by enabling the catalog mode. Then, you will provide the master with a project name that will be advertised to the default catalog server.

For Work Queue programs written in Python, you will use the specify_master_mode() API to set the catalog mode. Similarly, you will modify the program to specify a project name using the specify_name() API. An example of their usage is given below:

try:
   Q = WorkQueue(port = 0)
except:
   sys.exit(1)
Q.specify_master_mode(WORK_QUEUE_MASTER_MODE_CATALOG)
Q.specify_name("MYPROJECT")

For Work Queue programs in C/Perl, you will use the work_queue_specify_master_mode() API to set the catalog mode. To specify a project name, you will use work_queue_specify_name(). For example, a Work Queue program written in Perl will look like:

my $q = work_queue_create(0);
if (not defined($q)) {
   print "could not instantiate Work Queue master\n";
   exit 1;
}
work_queue_specify_master_mode($q, WORK_QUEUE_MASTER_MODE_CATALOG)
work_queue_specify_name($q, "MYPROJECT")

NOTE: Pick your own distinct name for MYPROJECT.

You will then start workers for your Work Queue master by specifying the option to use the default catalog server (-a option) and the project name of your master (-N option). Example: > work_queue_worker -d all -a -N MYPROJECT (Replace MYPROJECT with the distinct name you chose for the Work Queue master).

To successfully run Work Queue workers on multiple platforms, you may need to build and install CCTools on those platforms. Follow these instructions to download and install CCTools.

Problem 3

Apply Work Queue in your research.

Problem Statement

In completing Problem 1 and Problem 2, you have mastered the fundamentals of transforming and running large dynamic worfklows as Work Queue programs!

Now, can you think of one or two instances in your research where you can apply Work Queue to run computation/simulation workflows?