CCL Home Software Community Operations |
![]() TaskVine is a framework for building large scale data intensive dynamic workflows that run on high performance computing (HPC) clusters, GPU clusters, cloud service providers, and other distributed computing systems. A TaskVine application consists of a manager and a large number of worker processes. The application generates a large number of small tasks, which are distributed to workers. As tasks access external data sources and produce their own outputs, more and more data is pulled into local storage on cluster nodes. This data is used to accelerate future tasks and avoid re-computing exisiting results. The application gradually grows "like a vine" through the cluster. TaskVine is our third-generation workflow system, built on our twenty years of experience creating scalable applications in fields such as high energy physics, bioinformatics, molecular dynamics, and machine learning. Task Vine is currently a "tech preview" available from github for integration testing. We plan to make general user release via Conda in spring 2023.
Learn About TaskVineOnline Status Display![]() Publications(Showing papers with tag taskvine. See all papers instead.) |