Work Queue Tutorial
Review the start of the makeflow tutorial to
ensure that you have the software installed and the proper environment variables set.
Download the example Work Queue application in your favorite language:
C: wget http://ccl.cse.nd.edu/software/manuals/work_queue_example.c
Python: wget http://ccl.cse.nd.edu/software/manuals/work_queue_example.py
Perl: wget http://ccl.cse.nd.edu/software/manuals/work_queue_example.pl
Depending on your language, you will need to either compile the program (C) or set some environment variables (Python and Perl):
C: gcc work_queue_example.c -o work_queue_example -I${HOME}/cctools/include/cctools -L${HOME}/cctools/lib -lwork_queue -ldttools -lm
Python: export PYTHONPATH=${PYTHONPATH}:${HOME}/cctools/lib/python2.4/site-packages
Perl: export PERL5LIB=${PERL5LIB}:${HOME}/cctools/lib/perl5/site_perl
The example program simply runs gzip on whatever files you give on the command line, so run it like this:
C: ./work_queue_example *
Python: ./work_queue_example.py *
Perl: ./work_queue_example.pl *
The example program listens on the default port of 9123. So, you are likely to get the following error:
couldn't listen on port 9123: Address already in use
Whether you have this problem or not, go ahead and modify the program
to use port zero instead:
C Example:
q = work_queue_create(0);
if(!q) {
printf("couldn't listen....
return 1;
}
|
Python Example:
try:
q = WorkQueue(0)
except:
print "Instantiation of Work Queue failed!"
sys.exit(1)
|
Perl Example:
my $q = work_queue_create(0);
if (not defined($q)) {
print "Instantiation of Work Queue failed!\n";
exit 1;
}
|
Now, run it again, and you should see this:
listening on port XXXX...
...
waiting for tasks to complete...
Now, open up a new window and run a single local worker:
work_queue_worker localhost XXXX
Running Workers in SGE
Of course, we don't really want to run workers on the head node,
so let's instead start five workers using SGE. We will need
to use a project name to make this work, so first
edit the application and modify it to use a project name:
C Example:
q = work_queue_create(port);
if(!q) {
printf("couldn't listen....
return 1;
}
work_queue_specify_name(q,"MYPROJECT");
printf("listening on port %d...\n", work_queue_port(q));
|
Python Example:
try:
q = WorkQueue(port)
except:
print "Instantiation of Work Queue failed!"
sys.exit(1)
q.specify_name("MYPROJECT");
print "listening on port %d..." % q.port
|
Perl Example:
my $q = work_queue_create($port);
if (not defined($q)) {
print "Instantiation of Work Queue failed!\n";
exit 1;
}
work_queue_specify_project_($q,"MYPROJECT");
$port = work_queue_port($q);
print "listening on port $port...\n";
|
Now, submit workers to SGE with the same project name:
sge_submit_workers -N MYPROJECT 5
Creating worker submit scripts in dthain-workers...
Use the qstat command to observe that they are submitted to Torque:
job-ID prior name user state submit/start at queue
------------------------------------------------------------------------------------------------
18728 100.49976 worker.sh dthain r 06/02/2016 12:04:45 long@d6copt172.crc.nd.edu
18729 100.49976 worker.sh dthain r 06/02/2016 12:04:47 long@d6copt184.crc.nd.edu
18730 100.49976 worker.sh dthain r 06/02/2016 12:04:47 long@d6copt025.crc.nd.edu
18731 100.49976 worker.sh dthain r 06/02/2016 12:04:48 long@d6copt025.crc.nd.edu
18732 100.49976 worker.sh dthain r 06/02/2016 12:04:48 long@dqcneh084.crc.nd.edu
Now, restart your application and it will use the workers already running in SGE:
./work_queue_example *
listening on port XXXX...
You can leave the workers running there, if you want to start another application. They will remain until they have been idle for fifteen minutes, then will stop automatically.
In Class Exercise
Modify the example program to run each of the program sequentially.
That is, for each file to be compressed, run the task, wait for it
to finish, then run the next one, until all are complete.
|