|CCL HomeSoftware Community Operations||
Download and InstallationFirst log into the UA HPC center head node frost.hpc.arizona.edu by using ssh, Putty, or a similar tool. Once you have a shell, download and install the cctools software in your home directory in one of two ways:
export PATH=$HOME/cctools/bin:$PATHIf you use tcsh instead, then do this:
setenv PATH $HOME/cctools/bin:$PATHNow double check that you can run the various commands, like this:
makeflow -v work_queue_worker -v work_queue_status
Makeflow ExampleLet's being by using Makeflow to run a handful of simulation codes. First, make and enter a clean directory to work in:
cd $HOME mkdir tutorial cd tutorialNow, download this program, which performs a highly sophisticated simulation of black holes colliding together:
wget http://ccl.cse.nd.edu/software/tutorials/acic16/simulation.pyTry running it once, just to see what it does:
chmod 755 simulation.py ./simulation.py 5Now, let's use Makeflow to run several simulations. Create a file called example.makeflow and paste the following text into it:
input.txt: LOCAL /bin/echo "Hello Makeflow!" > input.txt output.1: simulation.py input.txt ./simulation.py 1 < input.txt > output.1 output.2: simulation.py input.txt ./simulation.py 2 < input.txt > output.2 output.3: simulation.py input.txt ./simulation.py 3 < input.txt > output.3 output.4: simulation.py input.txt ./simulation.py 4 < input.txt > output.4To run it on your local machine, one job at a time:
makeflow example.makeflow -j 1Note that if you run it a second time, nothing will happen, because all of the files are built:
makeflow example.makeflow makeflow: nothing left to doUse the -c option to clean everything up before trying it again:
makeflow -c example.makeflowArizona HPC uses the PBS batch system, so to run your jobs through PBS, do this: (Note that the -B option passes a bunch of details required by UA HPC.)
makeflow -T pbs -B "-W group_list=ericlyons -q windfall -ljobtype=serial -lselect=1:ncpus=1:mem=1gb -lpack:shared" example.makeflowIf you are working at another site that uses Condor or Torque or SGE, then you would invoke makeflow like this instead:
makeflow -T condor example.makeflow makeflow -T torque example.makeflow makeflow -T sge example.makeflow
Running Makeflow with Work QueueYou will notice that a workflow can run very slowly if you submit each job individually, because it typically takes 30 seconds or longer to start each batch job running. To get around this limitation, we provide the Work Queue system. This allows Makeflow to function as a master process that quickly dispatches work to remote worker processes.
makeflow -c example.makeflow makeflow -T wq example.makeflow -p 0 listening for workers on port XXXX. ...Now open up another shell and run a single worker process:
work_queue_worker frost.hpc.arizona.edu XXXXGo back to your first shell and observe that the makeflow has finished. Of course, remembering port numbers all the time gets old fast, so try the same thing again, but using a project name:
makeflow -c example.makeflow makeflow -T wq example.makeflow -N MYPROJECT listening for workers on port XXXX ...Now open up another shell and run your worker with a project name:
work_queue_worker -N MYPROJECT
Running Workers in PBSNote: This step doesn't work at UA HPC, due to some local firewall limitations, but the same technique works at other sites.
Of course, we don't really want to run workers on the head node, so let's instead use pbs_submit_workers to set up five workers in PBS for us:
pbs_submit_workers -N MYPROJECT 5 Creating worker submit scripts in dthain-workers... 2065026.i136 2065027.i136 2065028.i136 2065029.i136 2065030.i136Use the qstat command to observe that they are submitted to PBS:
qstat 2065027.i136 worker.sh dthain 0 R batch 2065028.i136 worker.sh dthain 0 R batch 2065029.i136 worker.sh dthain 0 R batch 2065030.i136 worker.sh dthain 0 R batchNow, restart your Makeflow and it will use the workers already running in PBS:
makeflow -c example.makeflow makeflow -T wq example.makeflow -N MYPROJECT listening for workers on port XXXX. ...You can leave the workers running there, if you want to start another Makeflow. They will remain until they have been idle for fifteen minutes, then will stop automatically.
If you add the -d all option to Makeflow, it will display debugging information that shows where each task was sent, when it was returned, and so forth:
makeflow -c example.makeflow makeflow -T wq example.makeflow -N MYPROJECT -d all listening for workers on port XXXX.
HomeworkNow you have everything needed to do the Homework Assignment.