About the CCL
We design software that enables our collaborators to easily harness large scale distributed systems such as clusters, clouds, and grids. We perform fundamental computer science research that enables new discoveries through computing in fields such as physics, chemistry, bioinformatics, biometrics, and data mining.CCL News and Blog
- A New Visualization Tool for TaskVine Released (04 Nov 2024)
- Integrating TaskVine with Merlin (16 Oct 2024)
- Introducing Shepherd: Simplifying Integration of Service Workflows into Task-Based Workflows (14 Oct 2024)
- TaskVine at ParslFest 2024 (07 Oct 2024)
- Predicting Resources of Tasks in Dynamic Workflows with Bucketing Algorithms at IPDPS 2024 (04 Mar 2024)
- TaskVine at the HEP Analysis Grand Challenge (04 Mar 2024)
- CCTools 7.8.0 released (27 Feb 2024)
- Distant Futures at SC 2023 (20 Nov 2023)
- Maximizing Data Utility at HPPSS/SC 2023 (20 Nov 2023)
- (more news)
Community Highlight
Makeflow is used to manage the data processing workflow of the Airborne Lidar Processing System (ALPS) for the Experimental Advanced Airborne Research Lidar (EAARL)
at the USGS. Over the course of several hours of flight time, about a
gigabyte of raw LIDAR waveform data is collected. This data must be
geo-referenced in order to convert it into point clouds broken into 2km
tiles suitable for GIS use. When we collect this data in the field, it
is critical that the field crew can process the data rapidly and take a
look at it for obvious problems so they can be corrected before the next
flight.
Using Makeflow, the data can be processed quickly on a
portable 32-core cluster in the field in about 20 minutes. The data can
be processed fast enough to do some cursory analysis and also
re-process it a few times if needed to troubleshoot issues. Using
Makeflow, it is easy to run the exactly same workflow in the field on
the portable cluster or back in the office on a multi-core system.
- David Nagle, US Geological Survey