Tutorial: Building Scalable Data Intensive Applications with Makeflow and Work Queue

Lecture and tutorial for the Applied CI Concepts course at the University of Arizona, September 11 and 13, 2012.

This short course will provide a brief introduction to writing scalable applications using Makeflow and Work Queue, developed by the Cooperative Computing Lab at the University of Notre Dame. The course modules consist of a lecture that gives an overview of the software and applications, a tutorial in how to install and use the tools on FutureGrid, and a homework assignment of more challenging tasks to be completed during the week.

Tuesday, September 11, 2012

Thursday, September 13, 2012

Software Resources

Additional Reading

(Showing papers with tag css2012. See all papers instead.)

Christopher Moretti, Andrew Thrasher, Li Yu, Michael Olson, Scott Emrich, and Douglas Thain, A Framework for Scalable Genome Assembly on Clusters, Clouds, and Grids, IEEE Transactions on Parallel and Distributed Systems, 23(12), December, 2012. DOI: 10.1109/TPDS.2012.80

Badi Abdul-Wahid, Li Yu, Dinesh Rajan, Haoyun Feng, Eric Darve, Douglas Thain, Jesus A. Izaguirre, Folding Proteins at 500 ns/hour with Work Queue, 8th IEEE International Conference on eScience (eScience 2012), October, 2012. DOI: 10.1109/eScience.2012.6404429

Michael Albrecht, Patrick Donnelly, Peter Bui, and Douglas Thain, Makeflow: A Portable Abstraction for Data Intensive Computing on Clusters, Clouds, and Grids, Workshop on Scalable Workflow Enactment Engines and Technologies (SWEET) at ACM SIGMOD, May, 2012. DOI: 10.1145/2443416.2443417

Dinesh Rajan, Anthony Canino, Jesus A Izaguirre, and Douglas Thain, Converting a High Performance Application to an Elastic Cloud Application, The 3rd IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2011), November, 2011.

Peter Bui, Dinesh Rajan, Badi Abdul-Wahid, Jesus Izaguirre, Douglas Thain, Work Queue + Python: A Framework For Scalable Scientific Ensemble Applications, Workshop on Python for High Performance and Scientific Computing (PyHPC) at the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (Supercomputing) , November, 2011.

Andrew Thrasher, Rory Carmichael, Peter Bui, Li Yu, Douglas Thain, and Scott Emrich, Taming Complex Bioinformatics Workflows with Weaver, Makeflow, and Starch, Workshop on Workflows in Support of Large Scale Science, pages 1-6, November, 2010. DOI: 10.1109/WORKS.2010.5671858