Makeflow Container Tutorial

For this tutorial, we will assume you have access to the XSEDE platform, specifically the Open Science Grid (OSG). If you do not, please speak with your campus champion or get in touch with someone at XSEDE. This tutorial should be read only after completing the Makeflow tutorial.

Downloading the Singularity Image

Login to the XSEDE single sign-on portal login.xsede.org using ssh, PuTTY, or a similar tool. Then, login to the Open Science Grid by running:

gsissh osg

Once you have a shell, we will enter the tutorial directory used in the Makeflow tutorial:

cd $HOME
cd tutorial

We will now pull the Singularity image we will use for this tutorial:

singularity pull docker://nekelluna/ccl_makeflow_examples
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/nekelluna/ccl_makeflow_examples:latest
Cache folder set to XXX/.singularity/docker
Importing: base Singularity environment
Exploding layer: sha256:22dc81ace0ea2f45ad67b790cddad29a45e206d51db0af826dc9495ba21a0b06.tar.gz
Exploding layer: sha256:1a8b3c87dba3ed16956c881a26eb0c40502c176a35767627d87557d16ea1e0df.tar.gz
Exploding layer: sha256:91390a1c435a20661a9e9afdaeb818638299a20d6ee1cc06bbcab8ae4d51994f.tar.gz
Exploding layer: sha256:07844b14977e91f82408cbb8932c7d6141d6ca8da7d6587b4d3065750d69186f.tar.gz
Exploding layer: sha256:b78396653dae2bc0d9c02c0178bd904bb12195b2b4e541a92cd8793ac7d7d689.tar.gz
Exploding layer: sha256:ce8a89f3b45265cfed3e3e1243d7801bac0233e6097423ea335de32f383625f1.tar.gz
Exploding layer: sha256:746d066a9bc84b9d1249e5ffb1e0c185ebc3186792bf0ee51470390357d1ed6e.tar.gz
Exploding layer: sha256:cf7ccc299a69a62fb94d3a800665cb3e2dfdde731198c05970ec13616e8cfee2.tar.gz
Exploding layer: sha256:beb3ec8652795ae8019047e87c72e97311e6a84ee5efb255c659c2b376bd4be1.tar.gz
Exploding layer: sha256:35444a90cc9acd0cb8020bfed633d61afbe9e5ef3c91858684fd196d536d1b70.tar.gz
Exploding layer: sha256:c6a9ef4b9995d615851d7786fbc2fe72f72321bee1a87d66919b881a0336525a.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: ./ccl_makeflow_examples.simg
Cleaning up...
Done. Container is at: ./ccl_makeflow_examples.simg

Use the -c option to clean everything up just to make sure we start from a blank slate:

makeflow -c example.makeflow

Running Makeflow with Singularity

Open Science Grid uses the Singularity container technology. If your compute site runs Docker, Makeflow has plenty of documentation to show you how to get up and running. For now, let us run the example makeflow from the previous tutorial with a Singularity image:

makeflow --singularity ccl_makeflow_examples.simg example.makeflow

If you would like to see further documentation on Singularity and Docker options for Makeflow, a good first stop is the help message in Makeflow:

makeflow --help

Running Makeflow with Singularity via Work Queue

To speed up our execution of the makeflow, we can use Work Queue workers as we saw in the previous tutorial. First, we should clean up our makeflow and then re-run with our submission type set to wq for Work Queue:

makeflow -c example.makeflow
makeflow -T wq -N OSG-makeflow -B '+ProjectName = "XX-XXXXXXXXX"' --singularity ./ccl_makeflow_examples.simg example.makeflow -p 0
listening for workers on port XXXX.
...

Now open up another shell and run a single worker process:

work_queue_worker -N OSG-makeflow

We will see the tasks complete one-by-one. Though we used the Singularity container technology for this tutorial, Makeflow also supports Docker. Please take a look at our documentation to find out more.

Collaborating with the Cooperative Computing Lab

Hopefully you have found this tutorial helpful! Try experimenting with writing your own makeflow using what you have learned. We would be happy to help you out! If you write a makeflow, please consider adding it to our ever-expanding Makeflow Examples repository on GitHub. There are a few criteria we would like you to meet for pull requests to this repository:

Your workflow must be specified in either the Make-style syntax or in JX.
All software dependencies (analysis tools, simulation codes, etc.) must be publicly available for download.
An in-depth README file to help other users run your makeflow must be included (look at some of the other workflows in the GitHub repository.

Once your pull request is made, we will continue to collaborate with you to meet your workflow needs! We hope to hear from you soon.