NBFlow is our newest CSSI project that brings together interactive notebook technologies (Jupyter), with reproducibility tools (SciUnit), and distributed workflows (TaskVine). The combined result will be a new system (NBFlow) that makes it easy to execute large scale workflows on thousands of cluster nodes, through the familiar interface of the executable notebook. While easily stated, this presents a number of challenges to reconcile the high throughput but unpredictable nature of clusters with the immediate feedback expected by notebooks. We will develop a number of new techniques to convert notebooks into workflows, execute them reliably on the cluster, and then return them back again to the user. This project is a collaboration with Kevin Lannon (U Notre Dame), Tanu Malik (DePaul U), and Shaowen Wang (U Illinois).
Related Publications
Declare, Resolve, Reuse: A Portable Data Layer for Notebook Workflows Across HPC Systems
Md Saiful Islam, Reyer Band, Jin Zhou, Tanu Malik, Kevin Lannon, and Douglas Thain
@inproceedings{floability-escience-2026,author={Islam, Md Saiful and Band, Reyer and Zhou, Jin and Malik, Tanu and Lannon, Kevin and Thain, Douglas},title={{Declare, Resolve, Reuse: A Portable Data Layer for Notebook Workflows Across HPC Systems}},booktitle={{IEEE Conference on eScience}},pages={9},year={2026},cclpaperid={1009},keywords={floability}}
Backpacks for Notebooks: Enabling Containerized Notebook Workflows in Distributed Environments
Md Saiful Islam, Talha Azaz, Raza Ahmad, A S M Shahadat Hossain, Furqan Baig, Shaowen Wang, Kevin Lannon, Tanu Malik, and Douglas Thain
@inproceedings{floability-escience-2025,author={Islam, Md Saiful and Azaz, Talha and Ahmad, Raza and Hossain, A S M Shahadat and Baig, Furqan and Wang, Shaowen and Lannon, Kevin and Malik, Tanu and Thain, Douglas},title={{Backpacks for Notebooks: Enabling Containerized Notebook Workflows in Distributed Environments}},booktitle={{IEEE Conference on eScience}},pages={9},year={2025},cclpaperid={1003},keywords={floability}}
