CCL Home

Research

Software Community Operations

Recent News in the CCL


Creating Better Force Fields on Distributed GPUs with Work Queue

ForceBalance is an open source software tool for creating accurate force fields for molecular mechanics simulation using flexible combinations of reference data from experimental measurements and theoretical calculations. These force fields are used to simulate the dynamics and physical properties of molecules in chemistry and biochemistry.

The Work Queue framework gives ForceBalance the ability to distribute computationally intensive components of a force field optimization calculation in a highly flexible way. For example, each optimization cycle launched by ForceBalance may require running 50 molecular dynamics simulations, each of which may take 10-20 hours on a high end NVIDIA GPU. While GPU computing resources are available, it is rare to find 50 available GPU nodes on any single supercomputer or HPC cluster. With Work Queue, it is possible to distribute the simulations across several HPC clusters, including the Certainty HPC cluster at Stanford, the Keeneland GPU cluster managed by Georgia Tech and Oak Ridge National Laboratories, and the Stampede supercomputer managed by the University of Texas. This makes it possible to run many simulations in parallel and complete the high level optimization in weeks instead of years.

 - Lee-Ping Wang, Stanford University Wed, 10 Dec 2014 15:15:00 +0000

CCTools 4.3 released

The Cooperative Computing Lab is pleased to announce the release of version 4.3.0 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, Weaver, DeltaDB, SAND, All-Pairs, and other software. This release has some important changes:
  • Peter Bui's Weaver is included. Weaver is a high level interface to Makeflow which allows to describe workflows using python. For more information see cctools/doc/man/weaver.1 and cctools/weaver/examples in the distribution.
  • This is also the first release to include DeltaDB, written by Peter Ivie and Douglas Thain. DeltaDB implements a model for time-varying schema-free data and underlies the query engine for the CCTools catalog server.
  • Backwards compatibility of master and workers pre-4.3 is broken. Workers from 4.3 cannot connect to masters pre-4.3, and masters from 4.3 will not accept connection from workers pre-4.3. The API did not change, thus unless you want to take advantage of new features, you should not need to modify your code.
  • The interface to work_queue_pool has been simplified, and all options have to be specified at the command line. Please see cctools/doc/man/work_queue_pool.1 for more information.
  • Undefined environment variables used in Makeflow are no longer allowed by the parser.
  • Binaries for 32bit architectures are not being distributed as part of this release. Please let us know if you need them.

Other highlights

  • [WorkQueue] Perl object oriented bindings have been added. See perldoc Work_Queue::Queue [B. Tovar]
  • [WorkQueue] A priority per task can now be specified. [D. Thain, B. Tovar]
  • [WorkQueue] --single-shot option added to workers to exit quickly after the master disconnects [D. Thain].
  • [WorkQueue] Hierarchy statistics when using foremen are now available. [B. Tovar, M. Wolf]
  • [WorkQueue] work_queue_pool code cleanup. [D. Thain, B. Tovar]
  • [Makeflow] New lexer and parser with cleaner semantics and error reporting. [B. Tovar]
  • [Parrot] Bug fix that allows parrot's temp-dir to be on GPFS. [P. Donnelly]
  • [Parrot] Several fixes to better support executables with threads. [P. Donnelly]
  • [Parrot] Update to use the newer ptrace API. [P. Donnelly]
  • [Parrot] Several updates to parrot_package_run. See cctools/doc/man/parrot_package_run. [H. Meng]
  • [Parrot] iRODS 4.x support. [D. Thain]

You can download the software here: cctools download

Thanks goes to the contributors and testers for this release: Peter Bui, Patrick Donnelly, Nick Hazekamp, Peter Ivie, Kangkang Li, Haiyan Meng, Peter Sempolinski, Douglas Thain, Ben Tovar, Lee-Ping Wang, Matthias Wolf, Anna Woodard, and Charles Zheng

Enjoy! Thu, 04 Dec 2014 19:37:00 +0000


Work Queue Powers Nanoreactor Simulations

Lee-Ping Wang at Stanford University, recently published a paper in Nature Chemistry describing his work in fundamental molecular dynamics.

The paper demonstrates the "nanoreactor" technique in which simple molecules are simulated over a long time scale to observe their reaction paths into more complex molecules.  For example, the picture below shows 39 Acetylene molecules merging into a variety of hydrocarbons over the course of 500ps simulated time.  This technique can be used to computationally predict reaction networks in historical or inaccessible environments, such as the early Earth or the upper atmosphere.

To compute the final reaction network for this figure, the team used the Work Queue framework to harness over 300K node-hours of CPU time on the Blue Waters supercomputer at NCSA.


Mon, 17 Nov 2014 20:18:00 +0000

Open Sourcing Civil Engineering with a Virtual Wind Tunnel

In addition to the CCL tools themselves, members of the CCL lab often collaborate with other research groups to help them solve their scientific problems, using collaborative computing. Often, such collaborative projects drive the development and debugging of our tools.

An uploaded design in the Virtual Wind Tunnel

One such project is a Virtual Wind Tunnel, which was created in collaboration with the Notre Dame Civil Engineering Department, as part of a larger project to explore collaboration in civil design. On the surface, this is a fairly simple idea. A user uploads a building shape for analysis to a web portal. Then, the user can run wind flow simulations upon horizontal cross sections of the building. Once complete, the results of these simulations can be viewed and downloaded.

Making all of this work, however, requires a large number of interlocking components. For now, I would just like to describe how the CCL tools play a role in this system. When simulations are to be run, one very simple way to deliver simulation tasks to available computing resources is to run a Work Queue worker on those machines. The front-end of the system runs a Work Queue master, which queues up tasks.

Viewing Results of a Simulation
This has several advantages, but the most important is that we can be flexible about the resources which we use at any given time, even using computing resources from multiple sources at the same time. For example, we have a small private cloud which we use for experimental purposes. We also have access to an on-campus SGE grid, but must share with many other customers. Our current approach is to set up a handful of VM on the private cloud, which run workers. If demand for simulations is high enough, we ask for more workers from the SGE.

By using Work Queue as a means of distributing tasks, we can be more flexible about the backend upon which those tasks are run. This allows us to tailor our resource usage to our actual needs and, as needed, to adjust our resource usage when appropriate. Mon, 01 Sep 2014 18:21:00 +0000

DeltaDB - A Scalable Database Design for Time-Varying Schema-Free Data

DeltaDB is a log-structure database and query model designed for time-varying and schema-free data. The following video gives a high level overview of DeltaDB and describes how the model is scalable using MapReduce.



This database design is implemented within CCTools in two parts. Part 1 (data storage) has been available for over a year and is called the catalog server. Part 2 (data analysis) has recently been implemented and is not yet in a release, but is available in the following commit:

https://github.com/pivie/cctools/commit/bca998baf00c71484b567110d73c36bd042c3b3e





The data model is designed to handle schema-free status reports from various services. And while the reports can be schema-free, most of the fields will normally remain the same between subsequent reports from the same instance of a service.






The first status report is saved in it's entirety, and then the subsequent reports are saved as changes (or "deltas") on the original report. Snapshots of the status of all services and instances are stored on a daily basis. This allows a query for analysis based on a given time frame to jump more quickly to the start of the time frame, rather than have to start at the very beginning of the life of the catalog server.




A query is performed by applying a series of operators to the data. For a distributed system, spatial distribution is when the data is distributed such that a given instance always ends up on the same node. In this situation, all but the last of the operators can be performed in the map stage of the MapReduce model. This allows for better scalability because less work has to be performed by a single node in the reduce stage.


Much more detail is provided in a paper which was published at IEEE Bigdata 2014, and is available at the following URL:

http://ccl.cse.nd.edu/research/papers/pivie-deltadb-2014.pdf

For further inquiries, please email pivie@nd.edu.


Mon, 18 Aug 2014 18:21:00 +0000

Packaging Applications with Parrot 4.2.0

CCTools 4.2.0 includes a new feature in Parrot that allows you to automatically observe all of the files used by a given application, and then collect them up into a self-contained package.  The package can then be moved to another machine -- even a different variant of Linux -- and then run correctly with all of its dependencies present. The created package does not depend upon Parrot and can be re-run in a variety of ways.
  
This article explains how to generate a self-contained package and then share it so that others can verify can repeat your applications. The whole process involves three steps: running the original application, creating the self-contained package, and the running the package itself.


Figure 1 Packaging Procedure
Step 1: Run the original program

Run your program under parrot_run and record the filename list and environment variables by using --name-list and --env-list parameters.

parrot_run --name-list namelist --env-list envlist /bin/bash
 
After the execution of this command, you can run your program inside parrot_run.  At the end of step 1, one file named namelist containing all the accessed file names and one file named envlist containing environment variables will be generated.  After everything is done, simple exit the shell.

Step 2: Generate a self-contained package

Use parrot_package_create to generate a package based on the namelist and envlist generated in step 1.

parrot_package_create --name-list namelist --env-path envlist --package-path /tmp/package
 
This command causes all of the files given in the name list to be copied into the package directory /tmp/package.  You may customize the contents of the package by editing the namelist or the package directory by hand.

Step 3: Repeat the program using the package

The newly created package is simply a complete filesystem tree that can be moved to any convenient location.  It can be re-run by any method that treats the package as a self-contained root filesystem.  This can be done by using Parrot again, by setting up a chroot environment, by setting up a Linux container, or by creating a virtual machine.

To run the package using Parrot, do this:

parrot_package_run --package-path /tmp/package /bin/bash 

To run the package using chroot, do this:

chroot_package_run --package-path /tmp/package /bin/bash

In both cases, you will be dropped into a shell in the preserved environment, where all the files used by the original command will be present.  You will definitely be able to run the original command -- whether you can run other programs depends upon the quantity of data preserved.

For more information, see these man pages:



Fri, 01 Aug 2014 19:18:00 +0000

CCTools 4.2.0 released

We are pleased to announce the release of version 4.2.0 of the Cooperative Computing Tools including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.
The software may be downloaded here: Download CCTools 4.2.0
This release is mostly a bug fix release, but introduces changes to the Work Queue protocol. Thus, workers from 4.2 do not work with masters pre 4.2.
Among the bug fixes and added capabilities are: Among the bug fixes and added capabilities are:
  • [General] Support for systemd log journal. (Patrick Donelly)
  • [WorkQueue] Several bug fixes (Douglas Thain. Dinesh Rajan, Ben Tovar)
  • [WorkQueue] Improvements to resource accounting. (Ben Tovar)
  • [WorkQueue] work_queue_graph_log, a script to plot Work Queue's log. (Ben Tovar)
  • [WorkQueue] Autosize option for workers to fill Condor slots. (Douglas Thain)
  • [WorkQueue] Added several example applications in apps/ (Dinesh Rajan)
  • [Chirp] Several bug fixes. (Patrick Donelly)
  • [Parrot] Package creation of accessed files for execution repeatability. (Haiyan Meng)
  • [Parrot] Correct mmap handling. (Patrick Donelly)
  • [Parrot] Fix linking to iRODS. (Patrick Donelly)
  • [Parrot] Option to disable CVMFS alien cache. (Ben Tovar)
  • [Parrot] Bug fixes targeting CVMFS. (Ben Tovar)
Thanks goes to the contributors for many features and bug fixes:
  • Jakob Blomer
  • Dan Bradley
  • Peter Bui
  • Patrick Donnelly
  • Nicholas Hazekamp
  • Peter Ivie
  • Haiyan Meng
  • Dinesh Rajan
  • Casey Robinson
  • Peter Sempolinski
  • Douglas Thain
  • Ben Tovar
  • Matthias Wolf
Please send any feedback to the CCTools discussion mailing list .


Thu, 31 Jul 2014 12:33:00 +0000

DeltaDB at IEEE BigData 2014

Peter Ivie will be presenting his work on the DeltaDB database model for time-varying schema-free data at the IEEE International Congress on Big Data in Anchorage. The DeltaDB concept is what underlies the query engine for the cctools catalog server.

Tue, 03 Jun 2014 16:50:00 +0000

Journal Paper on MAKER and Work Queue

Our paper on converting the MAKER bioinformatics analysis from MPI to Work Queue, done in collaboration with the Notre Dame Bioinformatics Laboratory was recently accepted for publication in the International Journal of Bioinformatics Research and Applications.

Tue, 03 Jun 2014 16:47:00 +0000

CCL Papers at CCGrid 2014

Three papers from the CCL were presented at the 2014 IEEE Conference on Cluster, Cloud, and Grid Computing in Chicago:
Tue, 03 Jun 2014 16:45:00 +0000

Toward a Common Model of Highly Concurrent Programming

(This is the short version of a talk I gave at the MTAGS workshop at Supercomputing 2013.  See the slides here.)

Historically, highly concurrent programming has been closely associated with high performance computing.  Two programming models have been dominant: shared memory machines in which concurrency was expressed via multiple threads, and distributed memory machines in which concurrency was expressed via explicit message passing.  It is widely agreed that both of these programming models are very challenging, even for the veteran programmer.  In both cases, the programmer is directly responsible for designing the program from top to bottom and handling all of the issues of granularity, consistency, and locality necessary to achieve acceptable performance, with very little help from the runtime or operating systems.

However, a new approach to concurrent programming has been emerging over the last several years, in which the user programs in a much higher level language and relies upon the system to handle many of the challenging underlying details.  To achieve this, the program is successively decomposed into simpler representations, such that each layer of the system can gradually adapt it to the hardware available.

The layers can be described as follows:
  • declarative language (DL) for compactly representing a complete program.
  • directed graph (DAG) to represent the expanded program and its resources.
  • bag of independent tasks (BOT) with explicit input and output dependencies.
  • A shared-nothing cluster to which data and tasks must be assigned.
Several different research communities have arrived at this computing model somewhat independently: the high performance computing community, the scientific workflow community, and the cloud computing community.  In each case, the scale and complexity of the systems in use eventually made it impossible for the programmer or the user to take responsibility for all of the challenges of parallel/distributed computing.  Although each community employs different technologies and has distinct optimization goals, the overall structure of these systems is surprisingly similar.

A (very incomplete) selection of systems that follow this model:


Layer Cloud Stack Workflow Stack HPC Stack
Declarative Language (DL)    Pig Weaver Swift-T
Directed Acyclic Graph (DAG)Map-Reduce    Makeflow -
Bag of Tasks (BOT)JobTracker Work Queue Master Turbine
Distributed Data HDFS Work Queue Workers    MPI

Each layer of the system fulfills a distinct need.  The declarative language (DL) at the top is compact, expressive, and easy for end users, but is intractable to analyze in the general case because it may have a high order of complexity, possibly Turing-complete.  The DL can be used to generate a (large) directed acyclic graph (DAG) that represents every single task to be executed.  The DAG is not a great user-interface language, but it is much more suitable for a system to perform capacity management and optimization because it is a finite structure with discrete components.  A DAG executor then plays the graph, dispatching individual tasks as their dependencies are satisfied.  The BOT consists of all the tasks that are ready to run, and is then scheduled onto the underlying computer, using the data dependencies made available from the higher levels.

Why bother with this sort of model?  It allows us to compare the fundamental capabilities and expressiveness of different kinds of systems.  For example, in the realm of compilers, everyone knows that a proper compiler consists of a scanner, a parser, an optimizer, and a code generator.  Through these stages, the input program is transformed from a series of tokens to an abstract syntax tree, an intermediate representation, and eventually to assembly code.  Not every compiler uses all of these stages, much less the same code, but by using a common language, it is helpful to understand, compare, and design new systems.


Mon, 19 May 2014 19:14:00 +0000

Visualizing 10,000 Cores in Condor

Our Condor pool at the University of Notre Dame has been slowly growing, in no small part due to our collaboration with the Center for Research Computing, where it is now scavenging unused cycles from HPC clusters at the CRC.  When the dedicated batch system leaves a node unused, Condor is started on that node and keeps going until the dedicated system wants the node back.  Depending on the time of year, that leaves anywhere between 4K and 10K nodes available in the Condor pool.

We have tried a number of approaches at visualizing this complex system over the years.  Our latest tool, the Condor Matrix Display started as a summer project by Nick Jaeger, a student from the University of Wisconsin at Eau Claire.  The display shows a colored bar for each slot in the pool, where the width is proportional to the number of cores.

With a quick glance, you can see how many users are busy and whether they are running "thin" (1 core) or "fat" (many core) jobs.  Sorting by the machine name gives you sense of how each sub-cluster in the pool is used:


While sorting by users gives you a sense of what users are dominating the pool:


The display is always a nice way of viewing the relatively new feature of "dynamic slot" in Condor.  A large multi-core machine is now represented as a single slot with multiple resources.  For example, this bit of the display shows a cluster of 8-core machines where some of the machines are unclaimed (green), some are running 4-core jobs (blue), and some are running 1-core jobs (green):



Fri, 14 Feb 2014 16:20:00 +0000

CCL Papers at Supercomputing 2013

Members of the CCL team presented several papers at Supercomputing 2013 in Denver, Colorado:



Mon, 18 Nov 2013 18:34:00 +0000

CCL Workshop 2013

The Annual CCL Workshop was held on October 10-11 at the University of Notre Dame. The CCL team ran beginning and advanced tutorials and gave highlights of the many new features and capabilities in the software. Community members spoke about the many ways in which Makeflow, Work Queue, Parrot, and Chirp are used to acclerate discoveries in fields such as bioinformatics, high energy physics, weather prediction, image analysis, and molecular dynamics. Everyone got together to share a meal, solve problems, and generate new ideas. Thanks to everyone who participated, and see you next year!
Mon, 14 Oct 2013 16:00:00 +0000

CCL Workshop October 10-11 at Notre Dame

Join us for the annual CCL workshop at the University of Notre Dame!

http://www.nd.edu/~ccl/workshop/2013

The workshop is an opportunity for beginners and experts alike to learn about the latest software release, get to know the CCL team, meet other people engaged in large scale scientific computing, and influence the direction of our research and development.

Thursday, October 10th
Afternoon: Introduction and Software Tutorials

Friday, October 11th
All Day: New Technology, Scientific Talks, and Discussion

There is no registration fee, however, space is limited, so please register in advance to reserve your place. We hope to see you there!




Fri, 11 Oct 2013 12:00:00 +0000

New Work Queue Paper at IEEE Cluster 2013

Michael Albrecht and Dinesh Rajan will present their latest work titled Making Work Queue Cluster Friendly for Data Intensive Scientific Applications.

In the original design of Work Queue, each worker was a sequential process that executed one task at a time. This paper describes the extension of Work Queue into two respects:

  • Workers can now run multiple tasks simultaneously, each sharing a local cache directory.
  • Workers can be combined into hierarchies, each headed by a foreman, which provides a common disk cache for each sub tree.

The effect of these two changes is to dramatically reduce the network footprint at the master process, and at each execution site. The resulting system is more 'friendly' to local clusters, and is capable of scaling to even greater sizes.
Wed, 21 Aug 2013 12:55:00 +0000

CCTools 4.0 Released

The Cooperative Computing Lab is pleased to announce the release of version 4.0 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.

This is the first release of the 4.0 series, with some major changes:
  • To support new features on WorkQueue, backwards compatibility of master and workers pre-4.0 is broken. Specifically, workers from 4.0 cannot connect to masters pre-4.0, and masters from 4.0 will not accept connection from workers pre-4.0. The API did not change, thus unless you want to take advantage of new features, you should not need to modify your code.

  • All code related to WorkQueue has been consolidated to its own library. When linking work queue applications in C, you will need to use: -lwork_queue -ldttools rather than just -ldttools. If you are using the perl or python bindings, no change is necessary.

  • The auto-mode option -a for communicating with the catalog server is being deprecated. It is now implied when a master, or project name (-M, -N) is specified.

  • Most tools now support long options at the command line (e.g., --help).
Highlights:

  • [WorkQueue] Support for workers hierarchies, with a master-foremen-workers paradigm. [Michael Albrecht]
  • [WorkQueue] Multi-slot workers. A worker now is able to handle more than one task at a time. [Michael Albrecht]
  • [WorkQueue] Resource reports. A worker reports its resources (disk, memory, and cpu) to the master, and each task in the master can specify a minimum of such resources. [Michael Albrecht, DeVonte Applewhite]
  • [WorkQueue] Authentication between master and workers when using the catalog server [Douglas Thain].
  • [WorkQueue] Python bindings now include most C API. [Dinesh Rajan]
  • [WorkQueue] Several bug fixes and code reorganization. [Dinesh Rajan, Michael Albrecht]
  • [WorkQueue] Policies can be specified to work_queue_pool to submit workers on demand. [Li Yu]
  • [Makeflow] Support for task categories. A rule can be labeled with a category, and required computational resources (disk, memory, and cpu) can be specified per category. Makeflow then automatically communicates these requirements to work queue or condor. [Ben Tovar]
  • [Parrot/Chirp] Support for a search system call added. Search allows for finding files in a number of directories with a shell pattern. See parrot_search for more information. [Patrick Donnelly, Brenden Kokoszka]
  • [Parrot] Several bug fixes for cvmfs support. [Douglas Thain, Ben Tovar, Patrick Donnelly]
  • [Monitor] A resource monitor/watchdog for computational resources (e.g. disk, memory, cpu, and io) that can be used standalone, or automatically by Makeflow and Work Queue. [Ben Tovar]
  • [Monitor] A visualizer that builds a webpage to show the resources histograms from the reports of the resource monitor. [Casey Robinson]


Please refer to the doc/ directory in the distribution for the usage of this new features. You can download the software here:
Download




Thanks goes to the contributors for this release: Michael Albrecht, DeVonte Applewhite, Peter Bui, Patrick Donnelly, Brenden Kokoszka, Kyle Mulholland, Francesco Prelz, Dinesh Rajan, Casey Robinson, Peter Sempolinski, Douglas Thain, Ben Tovar, and Li Yu.


Mon, 29 Jul 2013 12:11:00 +0000

Ph.D. Defense: Li Yu

Congratulations to Dr. Li Yu, who successfully defended his Ph.D. dissertation, Right-sizing Resource Allocations for Scientific Applications in Clusters, Grids, and Clouds!

Fri, 26 Jul 2013 12:05:00 +0000

Tutorial on Building Scalable Scientific Applications at XSEDE13

We will be offering a tutorial titled Building Scalable Scientific Applications using Makeflow and Work Queue as part of XSEDE 2013 in San Diego on July 22. Mon, 22 Jul 2013 12:00:00 +0000

Accelerating Protein Folding with Adaptive Weighted Ensemble and Work Queue

Computational protein folding has historically relied on long-running simulations of single molecules. Although many such simulations can run be at once, they are statistically likely to sample the same common configurations of the molecule, rather than exploring the many possible states it may have. To address this, a team of researchers from the University of Notre Dame and Stanford University designed a system that combined the Adaptive Weighted Ensemble technique to run thousands of short Gromacs and Protomol simulations in parallel with periodic resampling to explore the rich state space of a molecule. Using the Work Queue framework, these simulations were distributed across thousands of CPUs and GPUs drawn from the Notre Dame, Stanford, and commercial cloud providers. The resulting system effectively simulates the behavior of a protein at 500 ns/hour, covering a wide range of behavior in days rather than years.


- Jesus Izaguirre, University of Notre Dame and Eric Darve, Stanford University Sat, 01 Jun 2013 18:07:00 +0000

Dinesh Rajan Wins Best Talk at CCGrid 2013

Congratulations to CCL graduate student Dinesh Rajan, who won the Best Presentation Award at CCGrid 2013 for his presentation of
Case Studies in Designing Elastic Applications!

Tue, 21 May 2013 13:58:00 +0000

Tutorial on Makeflow and Work Queue at CCGrid 2013

Dinesh Rajan will present a tutorial on Building Elastic Applications with Makeflow and Work Queue as part of CCGrid 2013 in Delft, the Netherlands on May 13th. Come join us and learn how to write applications that scale up to hundreds or thousands of nodes running on clusters, clouds, and grids.


Fri, 22 Mar 2013 11:12:00 +0000

Elastic Apps Paper at CCGrid 2013

Dinesh Rajan will present his paper
Case Studies in Designing Elastic Applications at the IEEE International Conference on Clusters, Clouds, and Grids (CCGrid) in Delft, the Netherlands. This work was done in collaboration with Andrew Thrasher and Scott Emrich from the Notre Dame Bioinformatics Lab, and Badi Abdul-Wahid and Jesus Izaguirre from the Laboratory for Computational Life Sciences.


The paper describes our experience in designing three different elastic applications -- E-MAKER, Elastic Replica Exchange, and Folding at Work -- that run on hundreds to thousands of cores using the Work Queue framework. The paper offers six guidelines for designing similar applications:


  1. Abolish shared writes.
  2. Keep your software close and your dependencies closer.
  3. Synchronize two, you make company; synchronize
    three, you make a crowd.
  4. Make tasks of a feather flock together.
  5. Seek simplicity, and gain power.
  6. Build a model before scaling new heights.



Fri, 22 Mar 2013 10:40:00 +0000

Genome Assembly Paper in IEEE TPDS

A recent article in IEEE Transactions on Parallel and Distributed Computing describes our work in collaboration with the Notre Dame Bioinformatics Laboratory on SAND - The Scalable Assembler at Notre Dame.


In this article, we describe how to refactor the standard Celera genome assembly pipeline into a scalable computation that runs on thousands of distributed cores using the Work Queue. By explicitly handling the data dependencies between tasks, we are able to significantly improve runtime over Celera on a standard cluster. In addition this technique allows the user to break free of the shared filesystem and run on hundreds thousands of nodes drawn from clusters, clouds, and grids.



Thu, 21 Mar 2013 15:19:00 +0000


CCTools 3.7.0 Released!

The Cooperative Computing Lab is pleased to announce the release of version 3.7.0 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.



The software may be downloaded here.



This is a minor release which adds numerous features and fixes several bugs:




  • [WorkQueue] It is now possible to specify chunks (pieces) of an input file to be used as input for worker tasks. [Dinesh Rajan]

  • [Chirp] File extended attributes are now supported. [Patrick Donnelly]

  • [Makeflow] New -i switch now outputs pre-execution analysis of Makeflow DAG. [Li Yu]

  • [WorkQueue/Makeflow] Support for submitting tasks to the PBS batch submission platform added. [Dinesh Rajan]

  • [Makeflow] makeflow_log_parser now ignores comments in Makeflow logs. [Andrew Thrasher]

  • [Catalog] New catalog_update which reports information to a catalog server. [Peter Bui, Dinesh Rajan]

  • [WorkQueue] Various minor tweaks made to the API. [Li Yu, Dinesh Rajan]

  • [Catalog/WorkQueue] Support added for querying workers and tasks at run-time. [Douglas Thain]

  • [WorkQueue] Many environment variables removed in favor of option manipulation API. [Li Yu]

  • [Makeflow] Deprecated -t option (capacity tolerance) removed.

  • [WorkQueue] -W (worker status) now has working_dir and current_time fields.

  • [WorkQueue] -T (task status) now reports working_dir, current_time, address_port, submit_to_queue_time, send_input_start_time, execute_cmd_start_time. [Li Yu]

  • [WorkQueue] -Q (queue status) now reports working_dir.

  • [Makeflow] Input file (dependency) renaming supported with new "->" operator. [Michael Albrecht, Ben Tovar]

  • [WorkQueue] work_queue_pool now supports a new -L option to specify a log file. [Li Yu]

  • [WorkQueue] Tasks are now killed using SIGKILL.

  • [WorkQueue] Protocol based keep-alives added to workers. [Dinesh Rajan]



Thanks goes to the contributors for many minor features and bug fixes:




  • Michael Albrecht

  • Peter Bui

  • Patrick Donnelly

  • Brian Du Sell

  • Kyle Mulholland

  • Dinesh Rajan

  • Douglas Thain

  • Andrew Thrasher

  • Ben Tovar

  • Li Yu



Please send any feedback to the CCTools discussion mailing list.


Mon, 18 Feb 2013 16:29:00 +0000