Work Queue Tutorial

Review the start of the makeflow tutorial to ensure that you have the software installed and the proper environment variables set:

export CCTOOLS_HOME=/opt/cctools-7.0.19-x86_64-centos7
export PATH=${CCTOOLS_HOME}/bin:$PATH
export PYTHONPATH=${CCTOOLS_HOME}/lib/python2.7/site-packages:${PYTHONPATH}
export PERL5LIB=${CCTOOLS_HOME}/lib/perl5/site_perl/5.16.3:${PERL5LIB}
export TCP_LOW_PORT=6000
export TCP_HIGH_PORT=6100

cd $HOME
mkdir tutorial
cd tutorial

Download the example Work Queue application in your favorite language:

Python: wget http://ccl.cse.nd.edu/software/manuals/work_queue_example.py && chmod 755 work_queue_example.py
Perl:   wget http://ccl.cse.nd.edu/software/manuals/work_queue_example.pl && chmod 755 work_queue_example.pl
C:      wget http://ccl.cse.nd.edu/software/manuals/work_queue_example.c

If you are using python or perl, make sure that PYTHONPATH and PERL5LIB are set as show in makeflow tutorial. If you are using C, compile the program as:

 C: gcc work_queue_example.c \
    -o work_queue_example \
    -I${CCTOOLS_HOME}/include/cctools \
    -L${CCTOOLS_HOME}/lib \
    -lwork_queue -ldttools -lm -lz

The example program simply runs gzip on whatever files you give on the command line, so run it like this:

Python: ./work_queue_example.py *
Perl:   ./work_queue_example.pl *
C:      ./work_queue_example *

The example program listens on the default port of 9123. So, you may get the following error:

couldn't listen on port 9123: Address already in use

If you did not get the error, terminate the program typing control-c. Modify the program to use a port at random in the given ranges:

Python: 

try:
  q = WorkQueue(port = [6000,6100])
except:
  print("Instantiation of Work Queue failed!")
  sys.exit(1)

Perl: 

my $q = Work_Queue->new(port => [6000,6100]) || die "Instantiation of Work Queue failed! ($!)\n";

C: 

q = work_queue_create(0);
if(!q) {
  printf("couldn't create queue: %s\n", strerror(errno));
  return 1;
}

Note that for Perl and Python, we could use port = 0 as the environment variables TCP_LOW_PORT and TCP_HIGH_PORT were set when configuring CCTools for this tutorial.

Now, run it again, and you should see this: listening on port XXXX... ... waiting for tasks to complete...

Now, open up a new window and run a single local worker:

# You may need to reset your PATH:
# export CCTOOLS_HOME=/opt/cctools-7.0.19-x86_64-centos7
# export PATH=${CCTOOLS_HOME}/bin:$PATH

work_queue_worker localhost XXXX

Running Workers with Project Names

We will need to use a project name to make this work, so first edit the application and modify it to use a project name (change USERID to your Atmosphere user id):

Python: 

try:
    q = WorkQueue(port = [6000,6100], name = "MYPROJECT-USERID")
except:
    print "Instantiation of Work Queue failed!" 
    sys.exit(1)

Perl: 

my $q = Work_Queue->new(port => [6010,6100], name => "MYPROJECT-USERID")
  || die "Instantiation of Work Queue failed! ($!)\n";

C: 

q = work_queue_create(0);
if(!q) {
    printf("couldn't create queue: %s\n", strerror(errno));
	return 1;
}

work_queue_specify_name(q,"MYPROJECT-USERID");

printf("listening on port %d...\n", work_queue_port(q));

Now, run workers to with the same project name:

work_queue_worker -M MYPROJECT-USERID

# or you could type this at the HPC to launch 5 workers:
# pbs_submit_workers -p"-W group_list=ericlyons -q windfall -ljobtype=serial -lselect=1:ncpus=1:mem=1gb -lpack:shared" IP_OF_YOUR_INSTANCE PORT_WQ_IS_LISTENING 5
#
# or:
# pbs_submit_workers -M MYPROJECT-USERID 5 -p"-W group_list=ericlyons -q windfall -ljobtype=serial -lselect=1:ncpus=1:mem=1gb -lpack:shared"

Now, restart your application and it will use the workers already running:

./work_queue_example *
listening on port XXXX...

You can leave the workers running there, if you want to start another application. They will remain until they have been idle for fifteen minutes, then will stop automatically.

In Class Exercise

Modify the example program to run each of the program sequentially. That is, for each file to be compressed, submit the task, wait for it to finish, then run the next one, until all are complete.

Homework Assignment

If you finish all that, then move on to the Homework Assignment.