Tutorial: Writing Scalable Applications with Makeflow and Work Queue

This short course will provide a brief introduction to writing scalable applications using Makeflow and Work Queue, developed by the Cooperative Computing Lab at the University of Notre Dame. The course modules consist of a lecture that gives an overview of the software and applications, a tutorial in how to install and use the tools on the CRC, and a homework assignment of more challenging tasks to be completed during the week.

Friday, June 4th, 2016

More Resources

Additional Reading

(Showing papers with tag workqueue. See all papers instead.)

Ben Tovar, Ben Lyons, Kelci Mohrman, Barry Sly-Delgado, Kevin Lannon, and Douglas Thain, Dynamic Task Shaping for High Throughput Data Analysis Applications in High Energy Physics, IEEE International Parallel and Distributed Processing Symposium, June, 2022. DOI: 10.1109/IPDPS53621.2022.00041

Thanh Son Phung, Logan Ward, Kyle Chard, and Douglas Thain, Not All Tasks Are Created Equal: Adaptive Resource Allocation for Heterogeneous Tasks in Dynamic Workflows, WORKS Workshop on Workflows at Supercomputing, November, 2021.

Benjamin Tovar, Brian Bockelman, Michael Hildreth, Kevin Lannon, and Douglas Thain, Harnessing HPC resources for CMS jobs using a Virtual Private Network, 25th International Conference on Computing in High Energy and Nuclear Physics (CHEP), May, 2021. DOI: 10.1051/epjconf/202125102032

Tim Shaffer, Zhuozhao Li, Ben Tovar, Yadu Babuji, TJ Dasso, Zoe Surma, Kyle Chard, Ian Foster, and Douglas Thain, Lightweight Function Monitors for Fine-Grained Management in Large Scale Python Applications, IEEE International Parallel and Distributed Processing Symposium, May, 2021. DOI: 10.1109/IPDPS49936.2021.00088

Chao Zheng, Nathaniel Kremer-Herman, Tim Shaffer, and Douglas Thain, Autoscaling High Throughput Workloads on Container Orchestrators , IEEE Conference on Cluster Computing, pages 1-10, September, 2020. DOI: 10.1109/CLUSTER49012.2020.00024

Nick Hazekamp, Ben Tovar, and Douglas Thain, Dynamic Sizing of Continuously Divisible Jobs for Heterogeneous Resources, IEEE International Conference on e-Science, September, 2019. DOI: 10.1109/eScience.2019.00026

Nathaniel Kremer-Herman, Benjamin Tovar, and Douglas Thain, A Lightweight Model for Right-Sizing Master-Worker Applications, ACM/IEEE Supercomputing (SC), November, 2018. DOI: 10.1109/SC.2018.00042

Nicholas Hazekamp, Upendra Kumar Devisetty, Nirav Merchant, and Douglas Thain, MAKER as a Service: Moving HPC applications to Jetstream Cloud, IEEE International Conference on Cloud Engineering, pages 6, April, 2018. DOI: 10.1109/IC2E.2018.00029

Jeffrey Kinnison, Nathaniel Kremer-Herman, Douglas Thain, Walter Scheirer, SHADHO: Massively Scalable Hardware-Aware Distributed Hyperparameter Optimization, IEEE Winter Conference on Applications of Computer Vision, pages 1-10, March, 2018. DOI: 10.1109/WACV.2018.00086

Benjamin Tovar, Rafael Ferreira da Silva, Gideon Juve, Ewa Deelman, William Allcock, Douglas Thain, and Miron Livny, A Job Sizing Strategy for High-Throughput Scientific Workflows, IEEE Transactions on Parallel and Distributed Systems, 29(2), pages 240-253, February, 2018. DOI: 10.1109/TPDS.2017.2762310

Daniel (Yue) Zhang, Charles (Chao) Zheng, Dong Wang, Doug Thain, Chao Huang, Xin Mu, Greg Madey, Towards Scalable and Dynamic Social Sensing Using A Distributed Computing Framework, The 37th IEEE International Conference on Distributed Computing Systems (ICDCS 2017), June, 2017. DOI: 10.1109/ICDCS.2017.196

Dinesh Rajan and Douglas Thain, Designing Self-Tuning Split-Map-Merge Applications for High Cost-Efficiency in the Cloud, IEEE Transactions on Cloud Computing, 5(2), pages 303-316, April, 2017. DOI: 10.1109/TCC.2015.2415780

Matthias Wolf and Anna Woodard and Wenzhao Li and Kenyi Hurtado Anampa and Benjamin Tovar and Paul Brenner and Kevin Lannon and Mike Hildreth and Douglas Thain, Scaling Up a CMS Tier-3 Site with Campus Resources and a 100Gb/s Network Connection: What Could Go Wrong?, International Conference on Computing in High Energy Physics, October, 2016. DOI: 10.1088/1742-6596/898/8/082041

Peter Ivie and Douglas Thain, PRUNE: A Preserving Run Environment for Reproducible Computing, IEEE Conference on e-Science, October, 2016. DOI: 10.1109/eScience.2016.7870886

Anna Woodard, Matthias Wolf, Charles Mueller, Nil Valls, Ben Tovar, Patrick Donnelly, Peter Ivie, Kenyi Hurtado Anampa, Paul Brenner, Douglas Thain, Kevin Lannon and Michael Hildreth, Scaling Data Intensive Physics Applications to 10k Cores on Non-Dedicated Clusters with Lobster, IEEE Conference on Cluster Computing, September, 2015.

Charles (Chao) Zheng and Douglas Thain, Integrating Containers into Workflows: A Case Study Using Makeflow, Work Queue, and Docker, Workshop on Virtualization Technologies in Distributed Computing (VTDC), June, 2015. DOI: 10.1145/2755979.2755984

Anna Woodard, Matthias Wolf, Charles Nicholas Mueller, Ben Tovar, Patrick Donnelly, Kenyi Hurtado Anampa, Paul Brenner, Kevin Lannon, and Michael Hildreth, Exploiting Volatile Opportunistic Computing Resources with Lobster, Computing in High Energy Physics, January, 2015.

Badi Abdul-Wahid, Haoyun Feng, Dinesh Rajan, Ronan Costaouec, Eric Darve, Douglas Thain, and Jesus A. Izaguirre, AWE-WQ: Fast-Forwarding Molecular Dynamics using the Accelerated Weighted Ensemble, Journal of Chemical Information and Modeling, 54(10), pages 3033-3043, September, 2014. DOI: 10.1021/ci500321g

Andrew Thrasher, Zachary Musgrave, Brian Kachmark, Douglas Thain, and Scott Emrich, Scaling Up Genome Annotation with MAKER and Work Queue, International Journal of Bioinformatics Research and Applications, 10(4-5), pages 447-460, June, 2014. DOI: 10.1504/IJBRA.2014.062994

Olivia Choudhury, Nicholas L. Hazekamp, Douglas Thain, Scott Emrich, Accelerating Comparative Genomics Workflows in a Distributed Environment with Optimized Data Partitioning, C4BIO Workshop at IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), May, 2014.

Michael Albrecht, Dinesh Rajan, Douglas Thain, Making Work Queue Cluster-Friendly for Data Intensive Scientific Applications, IEEE International Conference on Cluster Computing, September, 2013. DOI: 10.1109/CLUSTER.2013.6702628

Dinesh Rajan, Andrew Thrasher, Badi Abdul-Wahid, Jesus A Izaguirre, Scott Emrich, and Douglas Thain, Case Studies in Designing Elastic Applications, 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May, 2013. DOI: 0.1109/CCGrid.2013.46

Christopher Moretti, Andrew Thrasher, Li Yu, Michael Olson, Scott Emrich, and Douglas Thain, A Framework for Scalable Genome Assembly on Clusters, Clouds, and Grids, IEEE Transactions on Parallel and Distributed Systems, 23(12), December, 2012. DOI: 10.1109/TPDS.2012.80

Badi Abdul-Wahid, Li Yu, Dinesh Rajan, Haoyun Feng, Eric Darve, Douglas Thain, Jesus A. Izaguirre, Folding Proteins at 500 ns/hour with Work Queue, 8th IEEE International Conference on eScience (eScience 2012), October, 2012. DOI: 10.1109/eScience.2012.6404429

Andrew Thrasher, Zachary Musgrave, Douglas Thain, Scott Emrich, Shifting the Bioinformatics Computing Paradigm: A Case Study in Parallelizing Genome Annotation Using Maker and Work Queue, IEEE International Conference on Computational Advances in Bio and Medical Sciences, February, 2012.

Peter Bui, Dinesh Rajan, Badi Abdul-Wahid, Jesus Izaguirre, Douglas Thain, Work Queue + Python: A Framework For Scalable Scientific Ensemble Applications, Workshop on Python for High Performance and Scientific Computing (PyHPC) at the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (Supercomputing) , November, 2011.

Dinesh Rajan, Anthony Canino, Jesus A Izaguirre, and Douglas Thain, Converting a High Performance Application to an Elastic Cloud Application, The 3rd IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2011), November, 2011.

Irena Lanc, Peter Bui, Douglas Thain, and Scott Emrich, Adapting Bioinformatics Applications for Heterogeneous Systems: A Case Study, Emerging Computational Methods for the Life Sciences Workshop at ACM HPDC, pages 7-13, June, 2011. DOI: 10.1145/1996023.1996025

Li Yu, Christopher Moretti, Andrew Thrasher, Scott Emrich, Kenneth Judd, and Douglas Thain, Harnessing Parallelism in Multicore Clusters with the All-Pairs, Wavefront, and Makeflow Abstractions, Journal of Cluster Computing, 13(3), pages 243-256, September, 2010. DOI: 10.1007/s10586-010-0134-7

Douglas Thain and Christopher Moretti, Abstractions for Cloud Computing with Condor, Syed Ahson and Mohammad Ilyas, Cloud Computing and Software Services: Theory and Techniques, pages 153-171, CRC Press, July, 2010. ISBN: 9781439803158

Peter Bui, Li Yu and Douglas Thain, Weaver: Integrating Distributed Computing Abstractions into Scientific Workflows using Python, Challenges of Large Applications in Distributed Environments at ACM HPDC 2010, June, 2010. DOI: 10.1145/1851476.1851570

Christopher Moretti, Michael Olson, Scott Emrich, and Douglas Thain, Scalable Modular Genome Assembly on Campus Grids, University of Notre Dame, Computer Science and Engineering Department, Technical Report 2009-04, July, 2009.

Li Yu, Christopher Moretti, Scott Emrich, Kenneth Judd, and Douglas Thain, Harnessing Parallelism in Multicore Clusters with the All-Pairs and Wavefront Abstractions, IEEE High Performance Distributed Computing, pages 1-10, June, 2009. DOI: 10.1145/1551609.1551613