Data Intensive Abstractions for High End Biometric Applications

Biometric research requires the execution of very large data intensive batch workloads. To evaluate new matching algorithms, researchers wish to compare thousands of images to each other by brute force. When these sort of workloads are submitted to conventional batch systems in the usual way, they induce massive amount ofs network and I/O traffic that result in very poor throughput. How can we execute such large workloads effectively?

To attack this problem, we are introducing data intensive abstractions that allow the user to easily provide the system with more information about the structure of a workload so that is can partition the data and execute it effectively. The abstraction explicitly specifies the data to be processed, the code that will process it, and the relationship between the two. One example of an abstraction is All-Pairs:

A computing system with an All-Pairs interface can easily find a more efficient implementation than a demand-paged filesystem. The input data can be staged to the computation nodes by a spanning tree, and the partitioning of work units into jobs can be done according to the performance properties of the system. In this project, we are designing a variety of similar data intensive abstractions that allow for the easy and efficient execution of large scientific workloads.

Publications

(Showing papers with tag abstractions. See all papers instead.)

Dinesh Rajan and Douglas Thain, Designing Self-Tuning Split-Map-Merge Applications for High Cost-Efficiency in the Cloud, IEEE Transactions on Cloud Computing, 5(2), pages 303-316, April, 2017. DOI: 10.1109/TCC.2015.2415780

Peter Bui, A Compiler Toolchain For Data Intensive Scientific Workflows, Ph.D. Thesis, University of Notre Dame, June, 2012.

Michael Albrecht, Patrick Donnelly, Peter Bui, and Douglas Thain, Makeflow: A Portable Abstraction for Data Intensive Computing on Clusters, Clouds, and Grids, Workshop on Scalable Workflow Enactment Engines and Technologies (SWEET) at ACM SIGMOD, May, 2012. DOI: 10.1145/2443416.2443417

Li Yu, Christopher Moretti, Andrew Thrasher, Scott Emrich, Kenneth Judd, and Douglas Thain, Harnessing Parallelism in Multicore Clusters with the All-Pairs, Wavefront, and Makeflow Abstractions, Journal of Cluster Computing, 13(3), pages 243-256, September, 2010. DOI: 10.1007/s10586-010-0134-7

Douglas Thain and Christopher Moretti, Abstractions for Cloud Computing with Condor, Syed Ahson and Mohammad Ilyas, Cloud Computing and Software Services: Theory and Techniques, pages 153-171, CRC Press, July, 2010. ISBN: 9781439803158

Peter Bui, Li Yu and Douglas Thain, Weaver: Integrating Distributed Computing Abstractions into Scientific Workflows using Python, Challenges of Large Applications in Distributed Environments at ACM HPDC 2010, June, 2010. DOI: 10.1145/1851476.1851570

Christopher Moretti, Abstractions for Scientific Computing on Campus Grids, Ph.D. Thesis, University of Notre Dame, June, 2010.

Hoang Bui, Peter Bui, Patrick Flynn and Douglas Thain, ROARS: A Scalable Repository for Data Intensive Scientific Computing, The Third International Workshop on Data Intensive Distributed Computing at ACM HPDC 2010, June, 2010. DOI: 10.1145/1851476.1851587

Christopher Moretti, Hoang Bui, Karen Hollingsworth, Brandon Rich, Patrick Flynn, and Douglas Thain, All-Pairs: An Abstraction for Data Intensive Computing on Campus Grids, IEEE Transactions on Parallel and Distributed Systems, 21(1), pages 33-46, January, 2010. DOI: 10.1109/TPDS.2009.49

Kyle Wheeler, Exploiting Locality with QThreads for Portable Parallel Performance, Ph.D. Thesis, University of Notre Dame, November, 2009.

Li Yu, Christopher Moretti, Scott Emrich, Kenneth Judd, and Douglas Thain, Harnessing Parallelism in Multicore Clusters with the All-Pairs and Wavefront Abstractions, IEEE High Performance Distributed Computing, pages 1-10, June, 2009. DOI: 10.1145/1551609.1551613

Christopher Moretti, Karsten Steinhaeuser, Douglas Thain, and Nitesh V. Chawla, Scaling Up Classifiers to Cloud Computers, IEEE International Conference on Data Mining (ICDM), pages 472-481, December, 2008. DOI: 10.1109/ICDM.2008.99

Brandon Rich and Douglas Thain, Poster: DataLab: Transactional Data Parallel Computing on an Active Storage Cloud, IEEE/ACM High Performance Distributed Computing, pages 233-234, June, 2008. DOI: 10.1145/1383422.1383461

Christopher Moretti, Jared Bulosan, Douglas Thain, and Patrick Flynn, All-Pairs: An Abstraction for Data Intensive Cloud Computing, IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 1-11, April, 2008. DOI: 10.1109/IPDPS.2008.4536311

Christopher Moretti, Jared Bulosan, Douglas Thain, and Patrick J. Flynn, Poster: All-Pairs: An Abstraction for Data Intensive Computing, IEEE/ACM Grid Computing, September, 2007.

Christopher Moretti, Timothy Faltemier, Douglas Thain, and Patrick J. Flynn, Challenges in Executing Data Intensive Biometric Workloads on a Desktop Grid, Workshop on Large Scale and Volatile Desktop Grids at IEEE IPDPS, pages 481-489, April, 2007. DOI: 10.1109/IPDPS.2007.370671

Douglas Thain, Sander Klous, Justin Wozniak, Paul Brenner, Aaron Striegel, and Jesus Izaguirre, Separating Abstractions from Resources in a Tactical Storage System, IEEE/ACM Supercomputing, pages 55-67, November, 2005. DOI: 10.1109/SC.2005.64

John Bent, Douglas Thain, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, and Miron Livny, Explicit Control in a Batch Aware Distributed File System, USENIX Networked Systems Design and Implementation (NSDI), pages 365-378, April, 2004.