Work Queue: A Scalable Manager/Worker Framework
Work Queue is a framework for building large manager-worker applications
that span thousands of machines drawn from clusters, clouds, and grids.
Work Queue applications are written in C, Perl, or Python using a simple API that allows users to
define tasks, submit them to the queue, and wait for completion.
Tasks are executed by a standard worker process that can run on
any available machine. Each worker calls home to the manager process,
arranges for data transfer, and executes the tasks.
The system handles a wide variety of failures, allowing
for dynamically scalable and robust applications.
Work Queue has been used to write applications that
scale from a handful of workstations up to tens of thousands
of cores running on supercomputers.
Parsl workflow system,
the Coffea analysis framework, the
the Makeflow workflow engine,
Accelerated Weighted Ensemble,
the SAND genome assembler,
and the All-Pairs and Wavefront abstractions.
The framework is easy to use, and has been used to teach courses in parallel computing,
cloud computing, distributed computing, and cyberinfrastructure at the University of Notre Dame,
the University of Arizona, and the University of Wisconsin - Eau Claire.
For More Information
Work Queue User's Manual
Work Queue API
( Python |
Work Queue Example Program
( Python |
Example Application Repository
Work Queue Status Display
Download Work Queue
Getting Help with Work Queue
Online Status Display
Online Introduction to Work Queue
(Showing papers with tag workqueue. See all papers instead.)
Tim Shaffer, Zhuozhao Li, Ben Tovar, Yadu Babuji, TJ Dasso, Zoe Surma, Kyle Chard, Ian Foster, and Douglas Thain,
, Lightweight Function Monitors for Fine-Grained Management in Large Scale Python Applications IEEE International Parallel & Distributed Processing Symposium, May, 2021.
Benjamin Tovar, Rafael Ferreira da Silva, Gideon Juve, Ewa Deelman, William Allcock, Douglas Thain, and Miron Livny,
, A Job Sizing Strategy for High-Throughput Scientific Workflows IEEE Transactions on Parallel and Distributed Systems, 29(2), pages 240-253, February, 2018. DOI: 10.1109/TPDS.2017.2762310
Daniel (Yue) Zhang, Charles (Chao) Zheng, Dong Wang, Doug Thain, Chao Huang, Xin Mu, Greg Madey,
, Towards Scalable and Dynamic Social Sensing Using A Distributed Computing Framework The 37th IEEE International Conference on Distributed Computing Systems (ICDCS 2017), June, 2017. DOI: 10.1109/ICDCS.2017.196
Anna Woodard, Matthias Wolf, Charles Mueller, Nil Valls, Ben Tovar, Patrick Donnelly, Peter Ivie, Kenyi Hurtado Anampa, Paul Brenner, Douglas Thain, Kevin Lannon and Michael Hildreth,
, Scaling Data Intensive Physics Applications to 10k Cores on Non-Dedicated Clusters with Lobster IEEE Conference on Cluster Computing, September, 2015.
Anna Woodard, Matthias Wolf, Charles Nicholas Mueller, Ben Tovar, Patrick Donnelly, Kenyi Hurtado Anampa, Paul Brenner, Kevin Lannon, and Michael Hildreth,
, Exploiting Volatile Opportunistic Computing Resources with Lobster Computing in High Energy Physics, January, 2015.
Badi Abdul-Wahid, Haoyun Feng, Dinesh Rajan, Ronan Costaouec, Eric Darve, Douglas Thain, and Jesus A. Izaguirre,
, AWE-WQ: Fast-Forwarding Molecular Dynamics using the Accelerated Weighted Ensemble Journal of Chemical Information and Modeling, 54(10), pages 3033-3043, September, 2014. DOI: 10.1021/ci500321g
Andrew Thrasher, Zachary Musgrave, Brian Kachmark, Douglas Thain, and Scott Emrich,
, Scaling Up Genome Annotation with MAKER and Work Queue International Journal of Bioinformatics Research and Applications, 10(4-5), pages 447-460, June, 2014. DOI: 10.1504/IJBRA.2014.062994
Dinesh Rajan, Andrew Thrasher, Badi Abdul-Wahid, Jesus A Izaguirre, Scott Emrich, and Douglas Thain,
, Case Studies in Designing Elastic Applications 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May, 2013. DOI: 0.1109/CCGrid.2013.46
Christopher Moretti, Andrew Thrasher, Li Yu, Michael Olson, Scott Emrich, and Douglas Thain,
, A Framework for Scalable Genome Assembly on Clusters, Clouds, and Grids IEEE Transactions on Parallel and Distributed Systems, 23(12), December, 2012. DOI: 10.1109/TPDS.2012.80
Badi Abdul-Wahid, Li Yu, Dinesh Rajan, Haoyun Feng, Eric Darve, Douglas Thain, Jesus A. Izaguirre,
, Folding Proteins at 500 ns/hour with Work Queue 8th IEEE International Conference on eScience (eScience 2012), October, 2012. DOI: 10.1109/eScience.2012.6404429
Peter Bui, Dinesh Rajan, Badi Abdul-Wahid, Jesus Izaguirre, Douglas Thain,
, Work Queue + Python: A Framework For Scalable Scientific Ensemble Applications Workshop on Python for High Performance and Scientific Computing (PyHPC) at the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (Supercomputing) , November, 2011.
Li Yu, Christopher Moretti, Andrew Thrasher, Scott Emrich, Kenneth Judd, and Douglas Thain,
, Harnessing Parallelism in Multicore Clusters with the All-Pairs, Wavefront, and Makeflow Abstractions Journal of Cluster Computing, 13(3), pages 243-256, September, 2010. DOI: 10.1007/s10586-010-0134-7