Tutorial: Building Scalable Data Intensive Applications with Makeflow and Work Queue

Lecture and tutorial for the Applied CI Concepts course at the University of Arizona, October 1 and 3, 2013.

This short course will provide a brief introduction to writing scalable applications using Makeflow and Work Queue, developed by the Cooperative Computing Lab at the University of Notre Dame. The course modules consist of a lecture that gives an overview of the software and applications, a tutorial in how to install and use the tools on FutureGrid, and a homework assignment of more challenging tasks to be completed during the week.

Tuesday, October 1st, 2013

Thursday, October 3, 2013

Software Resources

Additional Reading

(Showing papers with tag tutorial. See all papers instead.)

Michael Albrecht, Dinesh Rajan, Douglas Thain, Making Work Queue Cluster-Friendly for Data Intensive Scientific Applications, IEEE International Conference on Cluster Computing, September, 2013. DOI: 10.1109/CLUSTER.2013.6702628

Dinesh Rajan, Andrew Thrasher, Badi Abdul-Wahid, Jesus A Izaguirre, Scott Emrich, and Douglas Thain, Case Studies in Designing Elastic Applications, 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May, 2013. DOI: 0.1109/CCGrid.2013.46

Christopher Moretti, Andrew Thrasher, Li Yu, Michael Olson, Scott Emrich, and Douglas Thain, A Framework for Scalable Genome Assembly on Clusters, Clouds, and Grids, IEEE Transactions on Parallel and Distributed Systems, 23(12), December, 2012. DOI: 10.1109/TPDS.2012.80

Badi Abdul-Wahid, Li Yu, Dinesh Rajan, Haoyun Feng, Eric Darve, Douglas Thain, Jesus A. Izaguirre, Folding Proteins at 500 ns/hour with Work Queue, 8th IEEE International Conference on eScience (eScience 2012), October, 2012. DOI: 10.1109/eScience.2012.6404429

Michael Albrecht, Patrick Donnelly, Peter Bui, and Douglas Thain, Makeflow: A Portable Abstraction for Data Intensive Computing on Clusters, Clouds, and Grids, Workshop on Scalable Workflow Enactment Engines and Technologies (SWEET) at ACM SIGMOD, May, 2012. DOI: 10.1145/2443416.2443417

Dinesh Rajan, Anthony Canino, Jesus A Izaguirre, and Douglas Thain, Converting a High Performance Application to an Elastic Cloud Application, The 3rd IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2011), November, 2011.

Peter Bui, Dinesh Rajan, Badi Abdul-Wahid, Jesus Izaguirre, Douglas Thain, Work Queue + Python: A Framework For Scalable Scientific Ensemble Applications, Workshop on Python for High Performance and Scientific Computing (PyHPC) at the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (Supercomputing) , November, 2011.

Andrew Thrasher, Rory Carmichael, Peter Bui, Li Yu, Douglas Thain, and Scott Emrich, Taming Complex Bioinformatics Workflows with Weaver, Makeflow, and Starch, Workshop on Workflows in Support of Large Scale Science, pages 1-6, November, 2010. DOI: 10.1109/WORKS.2010.5671858