Building Scalable Elastic Applications using Makeflow and Work Queue
Tutorial at CCGrid
2013, May 13 2013, Delft University of Technology, The Netherlands. By
Dinesh Rajan and Prof. Douglas Thain.
This tutorial will provide an introduction to writing scalable
applications that are robust, elastic, and portable across different distributed
systems. We will use Makeflow and Work Queue, developed by the Cooperative Computing Lab, to build such
applications. These tools are used at Notre Dame and around the world to attack
large problems in fields such as chemistry, data mining, economics, biology,
physics, and more.
This tutorial is aimed at scientists, researchers, developers, graduate
students, and other participants interested in:
writing applications that transparently adapt and scale to the size of the
writing applications that are robust, fault-tolerant, and portable.
Familiarity with Unix
Ability to program in Python, Perl, or C (for the Work Queue portion of tutorial).
The tutorial will consist of a lecture and a hands-on instruction session. The
lecture will give an overview of the Makeflow and Work Queue software. The
hands-on instruction will demonstrate and walk through the installation and use
of Work Queue and Makeflow to write scalable programs. A set of practice
problems will also be provided to try on completion of the tutorial.
Makeflow Web Page
Work Queue Web Page
(Showing papers with tag tutorial. See all papers instead.)
Dinesh Rajan, Andrew Thrasher, Badi Abdul-Wahid, Jesus A Izaguirre, Scott Emrich, and Douglas Thain,
Case Studies in Designing Elastic Applications,
13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May, 2013. DOI: 0.1109/CCGrid.2013.46
Christopher Moretti, Andrew Thrasher, Li Yu, Michael Olson, Scott Emrich, and Douglas Thain,
A Framework for Scalable Genome Assembly on Clusters, Clouds, and Grids,
IEEE Transactions on Parallel and Distributed Systems, 23(12), December, 2012. DOI: 10.1109/TPDS.2012.80
Badi Abdul-Wahid, Li Yu, Dinesh Rajan, Haoyun Feng, Eric Darve, Douglas Thain, Jesus A. Izaguirre,
Folding Proteins at 500 ns/hour with Work Queue,
8th IEEE International Conference on eScience (eScience 2012), October, 2012. DOI: 10.1109/eScience.2012.6404429
Peter Bui, Dinesh Rajan, Badi Abdul-Wahid, Jesus Izaguirre, Douglas Thain,
Work Queue + Python: A Framework For Scalable Scientific Ensemble Applications,
Workshop on Python for High Performance and Scientific Computing (PyHPC) at the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (Supercomputing) , November, 2011.
Andrew Thrasher, Rory Carmichael, Peter Bui, Li Yu, Douglas Thain, and Scott Emrich,
Taming Complex Bioinformatics Workflows with Weaver, Makeflow, and Starch,
Workshop on Workflows in Support of Large Scale Science, pages 1-6, November, 2010. DOI: 10.1109/WORKS.2010.5671858