Container Management at IPDPS 2020

CCL grad student Tim Shaffer recently presented his recent work on container management at IPDPS 2020: Container technologies are seeing wider use at advanced computing facilities for managing highly complex applications that must execute at multiple sites. However, in a distributed high throughput computing setting, the unrestricted use of containers can result in the container explosion problem. If a new container image is generated for each variation of a job dispatched to a site, shared storage is soon exceeded. On the other hand, if a single large container image is used to meet multiple needs, the size of that container may become a problem for storage and transport. To address this problem, we observe that many containers have an internal structure generated by a structured package manager, and this information could be used to strategically combine and share container images. We develop LANDLORD to exploit this property and evaluate its performance through a combination of simulation studies and empirical measurement of high energy physics applications.

• Tim Shaffer, Nicholas Hazekamp, Jakob Blomer, and Douglas Thain, " Solving the Container Explosion Problem for Distributed High Throughput Computing " International Parallel and Distributed Processing Symposium, May, 2020.