CCL | Software | Install | Manuals | Forum | Papers
CCL Home


Software Community Operations
TaskVine Logo

TaskVine is a framework for building large scale data intensive dynamic workflows that run on high performance computing (HPC) clusters, GPU clusters, cloud service providers, and other distributed computing systems. A TaskVine application consists of a manager and a large number of worker processes. The application generates a large number of small tasks, which are distributed to workers. As tasks access external data sources and produce their own outputs, more and more data is pulled into local storage on cluster nodes. This data is used to accelerate future tasks and avoid re-computing exisiting results. The application gradually grows "like a vine" through the cluster. TaskVine is our third-generation workflow system, built on our twenty years of experience creating scalable applications in fields such as high energy physics, bioinformatics, molecular dynamics, and machine learning.

Task Vine is currently a "tech preview" available from github for integration testing. We plan to make general user release via Conda in spring 2023.

Developer Install
Tech Preview

Learn About TaskVine

  • TaskVine User's Manual
  • TaskVine Design Overview
  • TaskVine Python API | TaskVine C API
  • TaskVine Example Programs (Coming Soon)
  • TaskVine Status Display
  • Getting Help with TaskVine
  • Online Status Display


    (Showing papers with tag taskvine. See all papers instead.)