CCL Home

Software

Community

Operations

Makeflow Tutorial

For this tutorial, we will assume you have access to the XSEDE platform, specifically the Open Science Grid (OSG). If you do not, please speak with your campus champion or get in touch with someone at XSEDE.

Section 1

Download and Installation

Login to the XSEDE single sign-on portal login.xsede.org using ssh, PuTTY, or a similar tool. Then, login to the Open Science Grid by running:

gsissh osg

Once you have a shell, download and install the CCTools software in your home directory in one of two ways:

To install a binary package (faster):

wget http://ccl.cse.nd.edu/software/files/cctools-6.2.2-x86_64-redhat7.tar.gz
tar xvzf cctools-6.2.2-x86_64-redhat7.tar.gz
mv cctools-6.2.2-x86_64-redhat7 cctools

To build from source (slower):

git clone https://github.com/cooperative-computing-lab/cctools cctools-src
cd cctools-src
./configure --prefix $HOME/cctools
make
make install
cd $HOME

If you use bash then do this to set your path:

export PATH=$HOME/cctools/bin:$PATH

If you use tcsh instead, then do this:

setenv PATH $HOME/cctools/bin:$PATH

Now double check that you can run the various commands, like this:

makeflow -v
work_queue_worker -v
work_queue_status

Makeflow Example

Let us begin by using Makeflow to run a handful of simulation codes. First, make and enter a clean directory to work in:

cd $HOME
mkdir tutorial
cd tutorial

Now, download this program, which performs a highly sophisticated simulation of black holes colliding together:

wget http://ccl.cse.nd.edu/software/tutorials/makeflow/simulation.py

Try running it once, just to see what it does:

chmod 755 simulation.py
./simulation.py 5

Now, let's use Makeflow to run several simulations. Create a file called example.makeflow and paste the following text into it:

input.txt:
	LOCAL echo "Hello Makeflow!" > input.txt

output.1: simulation.py input.txt
	./simulation.py 1 < input.txt > output.1

output.2: simulation.py input.txt
	./simulation.py 2 < input.txt > output.2

output.3: simulation.py input.txt
	./simulation.py 3 < input.txt > output.3

output.4: simulation.py input.txt
	./simulation.py 4 < input.txt > output.4

To run it on your local machine, one job at a time:

makeflow example.makeflow -j 1

Note that if you run it a second time, nothing will happen, because all of the files are built:

makeflow example.makeflow
makeflow: nothing left to do

Use the -c option to clean everything up before trying it again:

makeflow -c example.makeflow

Running Makeflow with HTCondor

Open Science Grid uses the HTCondor batch system to schedule work to machines. Unlike other HTCondor implementations, they require a project name to bill for compute time. We will use the -B option to specify this additional parameter for OSG's HTCondor pool. Makeflow can directly submit to HTCondor like so:

makeflow -T condor -B '+ProjectName = "XX-XXXXXXXXX"' example.makeflow

If you are working at another site that uses SLURM or Torque or SGE, then you would invoke Makeflow like this instead:

makeflow -T slurm example.makeflow
makeflow -T torque example.makeflow
makeflow -T sge example.makeflow

There are plenty of other batch system which Makeflow supports. You can check to see if your preferred system is supported by running:

makeflow --help

Running Makeflow on Other Resources

Running Makeflow on Comet

First we need to login into Comet via login.xsede.org, this is done:

gsissh comet

First repeat the steps to download, install, and run Makeflow. Comet uses the SLURM batch system, so you can run the jobs through SLURM like this:

makeflow -T slurm -B "-p compute -t 60" example.makeflow

Running Makeflow on Bridges

First we need to login into Bridges via login.xsede.org, this is done:

gsissh bridges

First repeat the steps to download, install, and run Makeflow. Bridges uses the SLURM batch system, so you can run the jobs through SLURM like this:

makeflow -T slurm -B "-p RM-shared" example.makeflow

Even though both Comet and Bridges use the SLURM batch system, note the difference between the batch options passed in (the -B option). Every batch system works a little bit differently. Keep this in mind when running on private clusters or campus resources.

Section 2

Running Makeflow with Work Queue

You may notice that a workflow can run very slowly if you submit each job to the batch system individually. This is because it typically takes 30 seconds or longer to start each batch job. To get around this limitation, we provide the Work Queue execution engine. This allows Makeflow to function as a master process that quickly dispatches work to remote persistent worker processes. First, we should clean up our makeflow and then re-run with our submission type set to wq for Work Queue:

makeflow -c example.makeflow
makeflow -T wq example.makeflow -p 0
listening for workers on port XXXX.
...

Now open up another shell and run a single worker process, passing in the IP address where Makeflow is running and the port number printed out in the previous step:

work_queue_worker IP XXXX

Go back to your first shell and observe that the makeflow has finished. Of course, remembering port numbers all the time gets old fast, so try the same thing again, but using a project name:

makeflow -c example.makeflow
makeflow -T wq example.makeflow -N MYPROJECT
listening for workers on port XXXX
...

Now open up another shell and run your worker with a project name:

work_queue_worker -N MYPROJECT

Now, restart your makeflow and it will use the worker already running:

makeflow -c example.makeflow
makeflow -T wq example.makeflow -N MYPROJECT
listening for workers on port XXXX.
...

You can leave the workers running there if you want to start another makeflow. They will remain until they have been idle for too long (the default is fifteen minutes), then will clean up and shut down automatically.

If you add the -d all option to Makeflow, it will display debugging information that shows where each task was sent, when it was returned, and so forth:

makeflow -c example.makeflow
makeflow -T wq example.makeflow -N MYPROJECT -d all
listening for workers on port XXXX.

JSON and JX

Using the same makeflow as above, we're going to write the makelfow using JSON. Create a file called example.json and paste the following text into it:

{
"rules":[{
		"outputs":["input.txt"],
		"command":"echo \"Hello Makeflow!\" > input.txt",
		"local_job":true,
	},
	{
		"outputs":["output.1"],
		"inputs":["simulation.py","input.txt"],
		"command":"./simulation.py 1 < input.txt > output.1",
	},
	{
		"outputs":["output.2"],
		"inputs":["simulation.py","input.txt"],
		"command":"./simulation.py 2 < input.txt > output.2",
	},
	{
		"outputs":["output.3"],
		"inputs":["simulation.py","input.txt"],
		"command":"./simulation.py 3 < input.txt > output.3",
	},
	{
		"outputs":["output.4"],
		"inputs":["simulation.py","input.txt"],
		"command":"./simulation.py 4 < input.txt > output.4",
	}]
}

To run it on your local machine, one job at a time:

makeflow --json example.json -j 1

Use the -c option to clean everything up before trying it again:

makeflow -c --json example.json

Using the same makeflow as above, we're going to write the makelfow using JX. Create a file called example.jx and paste the following text into it:

{
"rules":[{
		"outputs":["input.txt"],
		"command":"echo \"Hello Makeflow!\" > input.txt",
		"local_job":true,
    },
    {
		"outputs":[format("output.%d",i)],
		"inputs":["simulation.py","input.txt"],
		"command":format("./simulation.py %d < input.txt > output.%d", i, i),
    } for i in range(1,5),
	],
}

To run it on your local machine, one job at a time:

makeflow --jx example.jx -j 1

Use the -c option to clean everything up before trying it again:

makeflow -c --jx example.jx

Now to leverage JX's ability to be used as a template, we will modify the existing example.jx to use a variable range. Change this line:

    } for i in range(1,5),

to:

    } for i in range(1,LIMIT),

To define what this limit is we can either pass it as a commandline option:

makeflow --jx example.makeflow --jx-define "LIMIT"=5

--or-- We can write it into a arguments file (args.jx) that configures the makeflow:

{
    "LIMIT"=10,
}

... and run it with:

makeflow --jx example.makeflow --jx-args args.jx

Resource Specifications

Eventually, we will need to specify how many resources our tasks need. This is because we would like to match our tasks with the workers which have the resources necessary to run them. To specify resources using the Make syntax, add this to the top of example.makeflow:

.MAKEFLOW CATEGORY analysis
.MAKEFLOW CORES 1
.MAKEFLOW MEMORY 100
.MAKEFLOW DISK 200

Now run it on your local machine, letting the resouces determine concurrency:

makeflow example.makeflow

You should be able to see all of the tasks running at the same time.

Use the -c option to clean everything up before trying it again:

makeflow -c example.makeflow

To see how resources can be used to limit the concurrency at larger scales, we will "scale up" this example by changing the number of expected cores.

.MAKEFLOW CORES 24

Try it again on your local machine:

makeflow example.makeflow

You should be able to see only two of the tasks running at the same time.

Use the -c option to clean everything up before trying it again:

makeflow -c example.makeflow

As an exercise, modify or add in a category to allow for three of the tasks to run at the same time.

Collaborating with the Cooperative Computing Lab

Hopefully you have found this tutorial helpful! Try experimenting with writing your own makeflow using what you have learned. We would be happy to help you out! If you write a makeflow, please consider adding it to our ever-expanding Makeflow Examples repository on GitHub. There are a few criteria we would like you to meet for pull requests to this repository:

Your workflow must be specified in either the Make-style syntax or in JX.
All software dependencies (analysis tools, simulation codes, etc.) must be publicly available for download.
An in-depth README file to help other users run your makeflow must be included (look at some of the other workflows in the GitHub repository.

Once your pull request is made, we will continue to collaborate with you to meet your workflow needs! We hope to hear from you soon.