CCL | Software | Download | Manuals | Forum | Papers
CCL Home

Research

Software Community Operations

Work Queue: A Scalable Manager/Worker Framework

Work Queue is a framework for building large manager-worker applications that span thousands of machines drawn from clusters, clouds, and grids. Work Queue applications are written in C, Perl, or Python using a simple API that allows users to define tasks, submit them to the queue, and wait for completion. Tasks are executed by a standard worker process that can run on any available machine. Each worker calls home to the manager process, arranges for data transfer, and executes the tasks. The system handles a wide variety of failures, allowing for dynamically scalable and robust applications.

Work Queue has been used to write applications that scale from a handful of workstations up to tens of thousands of cores running on supercomputers. Examples include the Parsl workflow system, the Coffea analysis framework, the the Makeflow workflow engine, Lobster, NanoReactors, ForceBalance, Accelerated Weighted Ensemble, the SAND genome assembler, and the All-Pairs and Wavefront abstractions. The framework is easy to use, and has been used to teach courses in parallel computing, cloud computing, distributed computing, and cyberinfrastructure at the University of Notre Dame, the University of Arizona, and the University of Wisconsin - Eau Claire.

For More Information

  • Work Queue User's Manual
  • Work Queue API (Python | Perl | C)
  • Work Queue Example Program (Python | Perl | C)
  • Example Application Repository
  • Work Queue Status Display
  • Download Work Queue
  • Getting Help with Work Queue
  • Online Status Display

    Online Introduction to Work Queue

    Publications

    (Showing papers with tag workqueue. See all papers instead.)