CCL | Software | Install | Manuals | Forum | Papers
CCL Home

Research

Software Community Operations 		Work Queue Logo

Work Queue is a framework for building large distributed applications that span thousands of machines drawn from clusters, clouds, and grids. Work Queue applications are written in Python, Perl, or C using a simple API that allows users to define tasks, submit them to the queue, and wait for completion. Tasks are executed by a general worker process that can run on any available machine. Each worker calls home to the manager process, arranges for data transfer, and executes the tasks. A wide variety of scheduling and resource management features are provided to enable the efficient use of large fleets of multicore servers. The system handles a wide variety of failures, allowing for dynamically scalable and robust applications.

Install Work Queue

Who Uses Work Queue?

Work Queue has been used to write applications that scale from a handful of workstations up to tens of thousands of cores running on supercomputers. Examples include the Parsl workflow system, the Coffea analysis framework, the the Makeflow workflow engine, SHADHO, Lobster, NanoReactors, ForceBalance, Accelerated Weighted Ensemble, the SAND genome assembler, and the All-Pairs and Wavefront abstractions. The framework is easy to use, and has been used to teach courses in parallel computing, cloud computing, distributed computing, and cyberinfrastructure at the University of Notre Dame, the University of Arizona, the University of Wisconsin, and many other locations.

Learn About Work Queue

  • Work Queue User's Manual
  • Work Queue API (Python | Perl | C)
  • Work Queue Example Program (Python | Perl | C)
  • Example Application Repository
  • Work Queue Status Display
  • Getting Help with Work Queue

    • Online Status Display

    Online Introduction to Work Queue

    Publications

    (Showing papers with tag workqueue. See all papers instead.)