The Chirp Distributed File System - Cooperative Computing Lab

CCL Home

Software

Community

Operations

Chirp is a user-level file system for collaboration across distributed systems such as clusters, clouds, and grids. Chirp allows ordinary users to discover, share, and access storage, whether within a single machine room or over a wide area network.

Chirp requires no special privileges. Unlike most standard file systems or storage services, Chirp does not require root access, kernel changes, special modules, or anything like that. It can be run by ordinary users to export ordinary file systems on any machine or port that you like.

Chirp is transparent. When used with Parrot or FUSE, Chirp servers can be transparently attached to existing ordinary applications -- like tcsh, vi, and perl -- without any sort of kernel changes or special privileges. Chirp is designed to give maximum compatibility with standard Unix semantics.

Chirp is easy to deploy. Chirp is designed to be deployed with a minimum of fuss. One simple command starts a Chirp server or a Chirp client. There is no complex configuration, installation, or setup to mess up. It just works. This makes Chirp ideal for on-the-fly storage management in batch computing and grid computing environments.

Documentation

Software and Systems

Publications

(Showing papers with tag chirp. See all papers instead.)

Patrick Donnelly and Douglas Thain,
Balancing push and pull in Confuga, an active storage cluster file system for scientific workflows,
Concurrency and Computation: Practice and Experience, 29(4), May, 2016. DOI: 10.1002/cpe.3834

Patrick Donnelly,
Data Locality Techniques in an Active Cluster Filesystem for Scientific Workflows,
Ph.D. Thesis, University of Notre Dame, April, 2016.

Patrick Donnelly, Nicholas Hazekamp, Douglas Thain,
Confuga: Scalable Data Intensive Computing for POSIX Workflows,
IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pages 392-401, May, 2015. DOI: 10.1109/CCGrid.2015.95

Patrick Donnelly and Douglas Thain,
Design of an Active Storage Cluster File System for DAG Workflows,
International Workshop on Data-Intensive Scalable Computing Systems, pages 37-42, November, 2013. DOI: 10.1145/2534645.2534656

Hoang Bui, Peter Bui, Patrick Flynn and Douglas Thain,
ROARS: A Robust Object Archival System for Data Intensive Scientific Computing,
Distributed and Parallel Databases, 30(5-6), Springer, August, 2012. DOI: 10.1007/s10619-012-7103-5

Patrick Donnelly and Douglas Thain,
Fine-Grained Access Control in the Chirp Distributed File System,
IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, May, 2012. DOI: 10.1109/CCGrid.2012.128

Douglas Thain, Michael Albrecht, Hoang Bui, Peter Bui, Rory Carmichael, Scott Emrich, and Patrick Flynn,
Data Intensive Computing with Clustered Chirp Servers,
Tevfik Kosar, Data Intensive Distributed Computing: Challenges and Solutions for Large Scale Information Management, pages 140-154, IGI, January, 2012. ISBN: 9781615209712

Patrick Donnelly, Peter Bui, Douglas Thain,
Attaching Cloud Storage to a Campus Grid Using Parrot, Chirp, and Hadoop,
IEEE International Conference on Cloud Computing Technology and Science, pages 488-495, November, 2010. DOI: 10.1109/CloudCom.2010.74

Rory Carmichael, Patrick Braga-Henebry, Douglas Thain, and Scott Emrich,
Biocompute: Toward a Collaborative Workspace for Data Intensive Bio-Science,
Workshop on Emerging Computational Methods for Life Sciences at ACM HPDC 2010, pages 489-498, June, 2010. DOI: 10.1145/1851476.1851547

Hoang Bui, Peter Bui, Patrick Flynn and Douglas Thain,
ROARS: A Scalable Repository for Data Intensive Scientific Computing,
The Third International Workshop on Data Intensive Distributed Computing at ACM HPDC 2010, June, 2010. DOI: 10.1145/1851476.1851587

T. Kosar, A. Hutanu, J. McLaren and D. Thain,
Coordination of Access to Large-scale Datasets in Distributed Environments,
A. Shoshani and D. Rotem, Scientific Data Management: Challenges, Existing Technology, and Deployment, Chapman and Hall / CRC Press, August, 2009. ISBN: 978-1420069808

Hoang Bui, Michael Kelly, Christopher Lyon, Mark Pasquier, Deborah Thomas, Patrick Flynn, and Douglas Thain,
Experience with BXGrid: A Data Repository and Computing Grid for Biometrics Research,
Journal of Cluster Computing, 12(4), pages 373, April, 2009. DOI: 10.1007/s10586-009-0098-7

Douglas Thain, Christopher Moretti, and Jeffrey Hemmes,
Chirp: A Practical Global Filesystem for Cluster and Grid Computing,
Journal of Grid Computing, 7(1), pages 51-72, March, 2009. DOI: 10.1007/s10723-008-9100-5

Paul Brenner, Justin Wozniak, Douglas Thain Aaron Striegel, Jeffrey Peng, and Jesus Izaguirre,
Biomolecular Committor Probability Calculation Enabled by Processing in Network Storage,
Journal of Parallel Computing, 34(11), pages 652-660, November, 2008. DOI: 10.1016/j.parco.2008.08.001

Brandon Rich and Douglas Thain,
Poster: DataLab: Transactional Data Parallel Computing on an Active Storage Cloud,
IEEE/ACM High Performance Distributed Computing, pages 233-234, June, 2008. DOI: 10.1145/1383422.1383461

Justin Wozniak, Paul Brenner, Douglas Thain, Aaron Striegel, and Jesus Izaguirre,
Making the Best of a Bad Situation: Prioritized Storage Management in GEMS,
Future Generation Computing Systems, 24(1), pages 10-16, January, 2008. DOI: 10.1016/j.future.2007.04.003

Douglas Thain and Christopher Moretti,
Efficient Access to Many Small Files in a Filesystem for Grid Computing,
IEEE Grid Computing, pages 243-250, September, 2007. DOI: 10.1109/GRID.2007.4354139

Christopher Moretti,
Flexible Object Based Filesystems for Scientific Computing,
M.S. Thesis, University of Notre Dame, April, 2007.

Sudharshan Vazhkudai, Douglas Thain, Xiaosong Ma, and Vincent Freeh,
Positioning Dynamic Storage Caches for Transient Data,
Workshop on High-Performance I/O Techniques and Deployment of Very Large Scale I/O Systems at IEEE Cluster Computing, pages 1-9, September, 2006. DOI: 10.1109/CLUSTR.2006.311900

Jeffrey Hemmes and Douglas Thain,
Cacheable Decentralized Groups for Grid Resource Access Control,
IEEE Grid Computing, pages 192-199, September, 2006. DOI: 10.1109/ICGRID.2006.311015

Douglas Thain,
Operating System Support for Space Allocation in Grid Storage Systems,
IEEE Grid Computing, pages 104-111, September, 2006. DOI: 10.1109/ICGRID.2006.311004

Douglas Thain, Christopher Moretti, Paul Madrid, Phil Snowberger, and Jeff Hemmes,
The Consequences of Decentralized Security in a Cooperative Storage System,
Workshop on Security in Storage at IEEE FAST, pages 82-94, December, 2005. DOI: 10.1109/SISW.2005.11

Douglas Thain, Sander Klous, Justin Wozniak, Paul Brenner, Aaron Striegel, and Jesus Izaguirre,
Separating Abstractions from Resources in a Tactical Storage System,
IEEE/ACM Supercomputing, pages 55-67, November, 2005. DOI: 10.1109/SC.2005.64

Justin Wozniak, Paul Brenner, Douglas Thain, Aaron Striegel, and Jesus Izaguirre,
Generosity and Gluttony in GEMS: Grid Enabled Molecular Simulations,
IEEE Symposium on High Performance Distributed Computing, pages 191-200, June, 2005. DOI: 10.1109/HPDC.2005.1520959