Work Queue Tutorial
Review the start of the makeflow tutorial to
ensure that you have the software installed and the proper environment variables set.
Download the example Work Queue application in your favorite language:
C: wget http://ccl.cse.nd.edu/software/manuals/work_queue_example.c
Python: wget http://ccl.cse.nd.edu/software/manuals/work_queue_example.py
Perl: wget http://ccl.cse.nd.edu/software/manuals/work_queue_example.pl
Depending on your language, you will need to either compile the program (C) or set some environment variables (Python and Perl):
C: gcc work_queue_example.c -o work_queue_example -I${HOME}/cctools/include/cctools -L${HOME}/cctools/lib -lwork_queue -ldttools -lm
Python: export PYTHONPATH=${PYTHONPATH}:${HOME}/cctools/lib/python2.4/site-packages
Perl: export PERL5LIB=${PERL5LIB}:${HOME}/cctools/lib/perl5/site_perl
The example program simply runs gzip on whatever files you give on the command line, so run it like this:
C: ./work_queue_example *
Python: ./work_queue_example.py *
Perl: ./work_queue_example.pl *
The example program listens on the default port of 9123. So, you are likely to get the following error:
couldn't listen on port 9123: Address already in use
Whether you have this problem or not, go ahead and modify the program
to use port zero instead:
C Example:
q = work_queue_create(0);
if(!q) {
printf("couldn't listen....
return 1;
}
|
Python Example:
try:
q = WorkQueue(0)
except:
print "Instantiation of Work Queue failed!"
sys.exit(1)
|
Perl Example:
my $q = work_queue_create(0);
if (not defined($q)) {
print "Instantiation of Work Queue failed!\n";
exit 1;
}
|
Now, run it again, and you should see this:
listening on port XXXX...
...
waiting for tasks to complete...
Now, open up a new window and run a single local worker:
work_queue_worker localhost XXXX
Running Workers in Condor
Of course, we don't really want to run workers on your computer,
so let's instead start five workers using Condor. We will need
to use a project name to make this work, so first
edit the application and modify it to use a project name:
C Example:
q = work_queue_create(0);
if(!q) {
printf("couldn't listen....
return 1;
}
work_queue_specify_name(q,"MYPROJECT");
printf("listening on port %d...\n", work_queue_port(q));
|
Python Example:
try:
q = WorkQueue(0)
except:
print "Instantiation of Work Queue failed!"
sys.exit(1)
q.specify_name("MYPROJECT");
print "listening on port %d..." % q.port
|
Perl Example:
my $q = work_queue_create(0);
if (not defined($q)) {
print "Instantiation of Work Queue failed!\n";
exit 1;
}
work_queue_specify_project_($q,"MYPROJECT");
$port = work_queue_port($q);
print "listening on port $port...\n";
|
Now, submit workers to Condor with the same project name:
condor_submit_workers -N MYPROJECT 5
Creating worker submit scripts in /tmp/nhazekam-workers...
Submitting job(s).....
5 job(s) submitted to cluster 643692.
Use the qstat command to observe that they are submitted to Torque:
condor_q -submitter $USER
-- Submitter: nhazekam@nd.edu : <129.74.85.211:9618?... : crcfe01.crc.nd.edu
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
643692.0 nhazekam 10/18 14:16 0+00:00:02 R 0 0.7 work_queue_worker
643692.1 nhazekam 10/18 14:16 0+00:00:02 R 0 0.7 work_queue_worker
643692.2 nhazekam 10/18 14:16 0+00:00:02 R 0 0.7 work_queue_worker
643692.3 nhazekam 10/18 14:16 0+00:00:02 R 0 0.7 work_queue_worker
643692.4 nhazekam 10/18 14:16 0+00:00:02 R 0 0.7 work_queue_worker
Now, restart your application and it will use the workers already running in Condor:
./work_queue_example *
listening on port XXXX...
You can leave the workers running there, if you want to start another application. They will remain until they have been idle for fifteen minutes, then will stop automatically.
Exercise
Modify the example program to run each of the program sequentially.
That is, for each file to be compressed, run the task, wait for it
to finish, then run the next one, until all are complete.
|