Building Scalable Scientific Applications using Makeflow and Work Queue

Tutorial at XSEDE13, July 22 2013, San Diego.

Introduction

This tutorial will provide an introduction to writing scalable scientific applications that are robust, elastic, and portable across different distributed systems. We will use Makeflow and Work Queue, developed by the Cooperative Computing Lab, to build such applications. These tools are used at Notre Dame and around the world to attack large problems in fields such as chemistry, data mining, economics, biology, physics, and more.

Target Audience

This tutorial is aimed at scientists, researchers, developers, graduate students, and other participants interested in:

Prerequisites

Tutorial Materials

The tutorial will consist of a lecture and a hands-on instruction session. The lecture will give an overview of the Makeflow and Work Queue software. The hands-on instruction will demonstrate and walk through the installation and use of Work Queue and Makeflow to write scalable programs. A set of practice problems will also be provided to try on completion of the tutorial.

  Makeflow

  Work Queue

  Demo

Software Resources

Additional Reading

(Showing papers with tag tutorial. See all papers instead.)

Michael Albrecht, Dinesh Rajan, Douglas Thain, Making Work Queue Cluster-Friendly for Data Intensive Scientific Applications, IEEE International Conference on Cluster Computing, September, 2013. DOI: 10.1109/CLUSTER.2013.6702628

Dinesh Rajan, Andrew Thrasher, Badi Abdul-Wahid, Jesus A Izaguirre, Scott Emrich, and Douglas Thain, Case Studies in Designing Elastic Applications, 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May, 2013. DOI: 0.1109/CCGrid.2013.46

Christopher Moretti, Andrew Thrasher, Li Yu, Michael Olson, Scott Emrich, and Douglas Thain, A Framework for Scalable Genome Assembly on Clusters, Clouds, and Grids, IEEE Transactions on Parallel and Distributed Systems, 23(12), December, 2012. DOI: 10.1109/TPDS.2012.80

Badi Abdul-Wahid, Li Yu, Dinesh Rajan, Haoyun Feng, Eric Darve, Douglas Thain, Jesus A. Izaguirre, Folding Proteins at 500 ns/hour with Work Queue, 8th IEEE International Conference on eScience (eScience 2012), October, 2012. DOI: 10.1109/eScience.2012.6404429

Michael Albrecht, Patrick Donnelly, Peter Bui, and Douglas Thain, Makeflow: A Portable Abstraction for Data Intensive Computing on Clusters, Clouds, and Grids, Workshop on Scalable Workflow Enactment Engines and Technologies (SWEET) at ACM SIGMOD, May, 2012. DOI: 10.1145/2443416.2443417

Dinesh Rajan, Anthony Canino, Jesus A Izaguirre, and Douglas Thain, Converting a High Performance Application to an Elastic Cloud Application, The 3rd IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2011), November, 2011.

Peter Bui, Dinesh Rajan, Badi Abdul-Wahid, Jesus Izaguirre, Douglas Thain, Work Queue + Python: A Framework For Scalable Scientific Ensemble Applications, Workshop on Python for High Performance and Scientific Computing (PyHPC) at the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (Supercomputing) , November, 2011.

Andrew Thrasher, Rory Carmichael, Peter Bui, Li Yu, Douglas Thain, and Scott Emrich, Taming Complex Bioinformatics Workflows with Weaver, Makeflow, and Starch, Workshop on Workflows in Support of Large Scale Science, pages 1-6, November, 2010. DOI: 10.1109/WORKS.2010.5671858