If you are transitioning from running code on your personal laptop to using a high-performance computing (HPC) cluster, you are about to encounter a major cultural shock: you do not have sudo privileges. On your own machine, installing a new tool is as simple as running sudo apt-get install, brew install, or clicking an installer. On a shared cluster, these commands will reward you with a blunt Permission denied. This restriction isn’t just bureaucratic cruelty; it is a fundamental safety measure. In a shared environment with hundreds of users, giving everyone root access would lead to broken configurations, security breaches, and software chaos.

So, how do you install the specific libraries, languages, and tools your research requires?

HPC systems use three primary tools for software management: Environment Modules, Conda, and Containers. Each has its own strengths, and using them correctly will save you days of troubleshooting—and keep you from accidentally drawing the ire of the cluster administrators.

Level 1: Environment Modules (The Built-In Way)

Before you try to download or compile anything yourself, you should check if the cluster administrators have already installed it for you. HPC systems use an environment management tool called Modules to manage hundreds of pre-installed, highly optimized software packages.

Because different researchers need different versions of the same software (e.g., Python 3.8 vs. Python 3.11, or different versions of CUDA), these packages sit dormant on the system until you explicitly ask for them.

Key Commands to Know:

• module avail: Displays a massive list of all software packages pre-compiled on the cluster. You can filter this by adding a keyword, such as module avail python or module avail gcc.
• module load <package>: Injects the chosen software into your current terminal session. For example, running module load python/3.10.8 updates your environment so that typing python points directly to that specific version.
• module list: Shows you which modules are currently active in your session.
• module purge: Clears out all loaded modules, giving you a clean slate. This is incredibly useful if two software packages are conflicting with one another.

Why you should use them:

Modules are pre-compiled specifically for the cluster’s underlying hardware. This means they are often optimized to run significantly faster than a generic version you download off the internet. Best of all, they take up exactly zero bytes of your personal storage quota. Always check module avail first.

Level 2: Conda / Miniconda (The User-Space Way)

What happens when you need a Python library or a niche scientific package that isn’t in the module list? For most data scientists and researchers, the immediate answer is Conda.

Conda allows you to create completely isolated “sandboxes” (environments) where you can install any package version you want entirely within your user account. However, Conda comes with two massive pitfalls that frequently break beginners’ workflows.

Pitfall #1: The Home Directory Trap

By default, when you run conda create or conda install, Conda downloads and extracts files into a hidden folder in your home directory (~/.conda or ~/.cache).

On almost all HPC clusters, your home directory (/home/username) has a strict, tiny storage quota (often between 10GB and 50GB) intended only for configuration files and source code. Because modern data science environments (especially those involving PyTorch or TensorFlow) can easily balloon to 10GB+ per environment, running a few installations can completely max out your storage quota. When this happens, you won’t even be able to log in or run basic commands.

The Fix: Redirect Conda to Storage or Scratch

Before installing anything with Conda, tell it to store its environments and package caches in your lab’s high-capacity group folder or a fast scratch directory. You can do this by creating or editing a configuration file named .condarc in your home directory:

# ~/.condarc
envs_dirs:
  - /scratch/username/conda/envs
  - /project/labname/shared_conda/envs
pkgs_dirs:
  - /scratch/username/conda/pkgs

Replace /scratch/username/ or /project/labname/ with the actual high-capacity data paths provided by your cluster administrators.

Pitfall #2: Compiling on the Login Node

When you log into a cluster, you land on a “login node.” This node is a shared hallway where hundreds of users are editing files and submitting jobs.

Running a heavy command like conda env create -f environment.yml fires up intense CPU and memory processes to resolve dependencies and compile code. Doing this on a login node will slow down the terminal experience for everyone else on the cluster and will usually trigger an automated system script that kills your process mid-installation.

The Fix: Use an Interactive Compute Session

Whenever you need to build or modify a Conda environment, request a temporary compute node using SLURM’s interactive mode:

# Request an interactive shell on a compute node for 1 hour
srun --tasks=1 --cpus-per-task=2 --mem=8G --time=01:00:00 --pty bash

Once your terminal transfers you to a compute node (e.g., user@node042), you can safely run your Conda installation commands without affecting anyone else.

Level 3: Containers (The Bulletproof Way)

Sometimes, even Conda isn’t enough. If your workflow requires a complex web of system libraries, an entirely different operating system (like a specific version of Ubuntu when the cluster runs Rocky Linux), or you want to replicate an exact software pipeline from a published paper, you need containers.

In the commercial tech world, Docker is the king of containers. However, Docker is fundamentally incompatible with shared HPC clusters because running Docker requires a background process (daemon) with root privileges. If you could run Docker on a cluster, you could easily bypass security and access other users’ data.

Apptainer / Singularity to the Rescue

To solve this, the HPC community developed Apptainer (formerly known as Singularity). Apptainer allows you to run containers completely in “user space” without needing sudo.

Instead of dealing with background daemons, Apptainer compresses an entire container environment into a single, portable file ending in .sif (Singularity Image Format).

Key Commands to Know:

1. Download and Convert a Docker Image: You can pull almost any public image directly from Docker Hub, and Apptainer will automatically translate it into an HPC-safe .sif file:

   apptainer pull ubuntu_python.sif docker://python:3.11-slim
   

2. Execute a Command Inside the Container: To run your code using the software isolated inside that container image, use exec:

   apptainer exec ubuntu_python.sif python3 my_script.py
   

3. Run an Interactive Shell Inside the Container: If you want to look around inside your containerized operating system environment:

   apptainer shell ubuntu_python.sif
   

Why you should use them:

Containers are the ultimate tool for scientific reproducibility. Once you find a container configuration that works, that .sif file will run exactly the same way on your university cluster, a national lab supercomputer, or a cloud instance five years from now.

Summary: The Software Management Checklist

To keep your account active and the cluster running smoothly, run through this mental checklist every time you need new software:

1. Check module avail first. If it’s already installed by the system administrators, load it and move on.
2. Check your storage quotas. Run your cluster’s quota check command (like myquota or lfs quota) to ensure your home directory isn’t on the verge of filling up.
3. Configure your .condarc file. Never leave Conda on its default settings; always point your environment and package directories to a high-capacity scratch or group folder.
4. Never install software on a login node. Use srun to spin up an interactive compute session before compiling or running extensive installers.
5. Use Apptainer for complex pipelines. If a workflow requires intricate system dependencies or absolute reproducibility, skip the installation headaches entirely and look for a container image.

When you log into a university or national lab cluster, you aren’t logging directly into the powerful machines that run your code. Instead, you land on a “login node” shared with dozens of other users. To actually run your heavy computations, you have to request permission from a gatekeeper.

On most High-Performance Computing (HPC) systems today, that gatekeeper is SLURM (Simple Linux Utility for Resource Management). Think of SLURM as a reservation system for a supercomputer. While it has a reputation for being complex, most day-to-day work comes down to a handful of basic commands. This guide covers the essential subset you need to get started: submitting jobs, monitoring them, stopping them when things go sideways, and understanding why they might be stuck in line.

One major caveat: SLURM is highly configurable, and every institution customizes it differently. Queue names, time limits, memory defaults, and specific resource rules vary from cluster to cluster. The commands here are standard SLURM, but you should always cross-reference them with your local site’s documentation.

Submitting a Job

You don’t run heavy scripts directly in the terminal on an HPC. Instead, you wrap your commands in a shell script and hand it over to SLURM using the sbatch command:

sbatch my_job.sh

When SLURM accepts your job, it assigns it a unique Job ID and prints it to the screen:

Submitted batch job 847312

You will use this number to track, manage, or cancel your job. A minimal, beginner-friendly job script looks like this:

#!/bin/bash
#SBATCH --job-name=hpc-demo
#SBATCH --output=hpc-demo-%j.out
#SBATCH --error=hpc-demo-%j.err
#SBATCH --time=04:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G

# The commands below will run on the compute node once allocated:
echo "Starting my analysis..."
python3 my_analysis_script.py --input data.csv

The special #SBATCH lines at the top tell SLURM exactly what resources your code needs to run.

What do these headers mean?

• --job-name: A friendly nickname for your job so you can spot it easily in the queue.
• --output and --error: Where your program’s text output and error messages should be saved. The %j is a variable that automatically fills in your unique Job ID, preventing different runs from overwriting each other’s logs.
• --time: The maximum runtime allowed (Hours:Minutes:Seconds). If your job exceeds this, SLURM will stop it automatically.
• --cpus-per-task and --mem: The requested number of CPU cores and total RAM (8 Gigabytes in this case).

Passing Arguments: Header vs. Command Line

Every #SBATCH line inside your script is identical to passing a flag directly to sbatch on the command line. These two methods accomplish the exact same thing:

# Option A: All configuration is kept inside the script header
sbatch my_job.sh

# Option B: Overriding the script's defaults from the command line
sbatch --time=08:00:00 --mem=16G my_job.sh

Command-line flags always take precedence over header directives. The best habit to build is putting your sensible defaults in the script header—the values you expect to use 90% of the time—and then overriding them on the command line for one-off experiments that require extra memory or longer runtimes.

Watching Your Jobs

Once you submit a job, squeue is the command you will use most often. Running it without arguments shows every single job currently running on the entire cluster, which is overwhelming. Instead, filter it to show only your jobs:

squeue -u $USER

The output will look something like this:

JOBID    PARTITION  NAME        USER     ST   TIME   NODES  NODELIST(REASON)
847312   general    hpc-demo    dthain   R    0:42   1      node047
847313   general    hpc-demo    dthain   PD   0:00   1      (Resources)
847314   general    hpc-demo    dthain   PD   0:00   1      (Priority)

The ST (State) column tells you what your job is currently doing:

• R (Running): Your job is actively executing on a compute node.
• PD (Pending): Your job is waiting in line.

The NODELIST(REASON) column explains why a pending job hasn’t started yet. (Resources) means the cluster is currently full and waiting for other users’ jobs to finish. (Priority) means your job is sitting behind higher-priority work in the queue.

If you want to check when SLURM expects your job to start, add the --start flag:

squeue -u $USER --start

Note: SLURM’s start-time estimates are notoriously optimistic, as they assume every running job will use its maximum requested time. Treat these numbers as rough ballparks rather than guarantees.

Cancelling a Job

If you realize your script has a bug, or you accidentally requested the wrong parameters, you can clear it out of the system using scancel and its Job ID:

scancel 847312

To wipe the slate clean and cancel all of your jobs at once:

scancel -u $USER

If you only want to clear out your waiting jobs while letting your active, running jobs finish safely, specify the pending state:

scancel -u $USER -t pending

Checking Cluster Availability

Before you submit a massive job, it is helpful to see how busy the cluster is. The sinfo command provides a snapshot of the available hardware pools (called “partitions”):

sinfo

PARTITION  AVAIL  TIMELIMIT   NODES  STATE   NODELIST
general*   up     7-00:00:00  12     idle    node[001-012]
gpu        up     2-00:00:00  4      mix     gpu[01-04]
debug      up     0-01:00:00  2      idle    node[013-014]

Pay attention to the STATE column:

• idle: The nodes are completely free and ready for work.
• mix: Some CPU cores on these nodes are busy, but others are still available.
• alloc: The nodes are completely full.
• drain: The nodes have been taken offline by administrators for maintenance.

Inspecting a Live or Stuck Job

For detailed troubleshooting on a specific job, scontrol show job will output a comprehensive diagnostic record:

scontrol show job 847312

The output is quite long, but look closely for JobState, Reason, and TRES (the exact resource request). If a job has been pending for days, the Reason field will often reveal if it is blocked by an administrative hold, an impossible resource request, or standard queue traffic.

Checking What Completed Jobs Actually Used

One of the biggest mistakes beginners make is overestimating how much memory or time their code needs. Requesting too many resources forces your job to wait in line much longer than necessary.

To look back at a completed job and see its actual resource footprint, use sacct:

sacct -j 847312 --format=JobID,JobName,State,Elapsed,MaxRSS,ReqMem

The MaxRSS field shows the peak amount of RAM your job actually consumed during its run. Compare this against ReqMem (what you requested). If your job only used 2GB of RAM but you requested 32GB, you are artificially delaying your own queue times. Aim to request roughly 20–30% more memory than your historical peak to keep your queue times short while avoiding out-of-memory errors.

A Few Silent Pitfalls to Avoid

SLURM enforces local cluster rules strictly, and it often does so silently. Keep these four items in mind as you review your institution’s specific user guide:

• Partition Time Caps: Most cluster queues have hard maximum runtimes (e.g., a 24-hour limit). If you ask for 48 hours on a 24-hour queue, SLURM will reject your job the second you attempt to submit it.
• Account Flags: Many institutions require you to specify a billing project or account via --account=my_lab_group. If you omit this, your job may fail instantly or be assigned to a lowest-priority pool.
• Silent Memory Defaults: If you don’t specify a --mem flag, SLURM will assign you a default value per core. On some clusters, this default can be incredibly small (like 1GB), causing your program to crash unexpectedly with an OUT_OF_MEMORY error. Always declare your memory explicitly.
• Job Arrays for Repetitive Work: If you need to run the exact same analysis script over 50 different data files, do not run sbatch 50 separate times. Instead, look into Job Arrays using sbatch --array=1-50. It allows the scheduler to handle your workload as a single cohesive unit, saving your sanity and keeping the cluster running smoothly.

Debugging distributed systems on HPC has its own rhythm. The feedback loop is slow, the machines are shared, and the error you are chasing may only appear under load, across nodes, or after a scheduler delay you cannot reproduce locally. This note grew out of issue #4415: a segmentation fault (SIGSEGV) in task_min_resources inside work_queue.c, triggered by Makeflow running a large parallel workflow in Work Queue mode.

The habits below are less about any single tool and more about discipline: keeping your changes visible, your experiments small, and your teammates in the loop. They apply equally whether you are working in work_queue.c or anywhere else in a large C codebase running on a shared cluster.

Talk to your teammates early and often

The most underrated debugging tool is a colleague. When you hit a wall the instinct is to keep grinding alone until you have something to show. Resist it. A two-sentence description of what you changed and where you are stuck surfaces assumptions you did not know you were making before you spend a day looking in the wrong function.

Post a short note to your team channel whenever you switch gears. If you are about to run a batch of SLURM jobs to test a hypothesis, say so. Someone may have already run that experiment, or may know that the catalog tick interval is configurable and could be shortened for testing.

The flip side: when someone asks for help, give them the full picture up front. Paste the Valgrind output, the backtrace, and a short description of what you have already tried. “Here is what I know” cuts the back-and-forth in half.

Put your changes somewhere others can see them

Debugging on HPC usually means modifying source, rebuilding, and staging a binary on a shared login node where the next person has no idea what you changed. The fix is obvious but easy to skip when you are in a hurry: commit and push early, to a branch, a fork, or a scratch repository your colleagues can actually reach.

git checkout -b debug/task-min-resources-segfault
# edit work_queue/src/work_queue.c
git add work_queue/src/work_queue.c
git commit -m "add null check before task_min_resources dereferences ready list node"
git push origin debug/task-min-resources-segfault

A pushed branch does several things at once. It gives collaborators a URL to look at instead of a terminal paste. It lets them check out your exact tree, reproduce the build, and point out something you missed. And it is an automatic checkpoint: if make clean goes sideways or a node corrupts your scratch directory, the work is not gone. Uncommitted changes on a cluster login node are one bad disk event away from disappearing.

Keep a running log of what you changed and why

Segfault hunts often take days or weeks. Memory is not reliable across that span. Keep a plain text file, a section in the PR description, or a lab notebook — and update it every time you try something:

2026-06-09  opened #4415; Valgrind log attached; fault site is work_queue.c:8174 in task_min_resources
2026-06-10  added null check on q->ready_list head before dereferencing — crash still appears on run 3 of 5
2026-06-11  hypothesis: node is removed from the ready list but list_next still returns it; added fprintf around list traversal
2026-06-12  confirmed: stale pointer survives across catalog tick; looking at where nodes are freed vs unlinked

This log is not for posterity. It is for you, tomorrow morning, when you cannot remember why you commented out that block. It is also what you paste when you ask a teammate for help.

Scale down before you scale out

Submitting a five-hundred-task Makeflow workflow to test a one-line fix is one of the most reliable ways to waste an afternoon. Queue wait times are unpredictable, the logs from hundreds of workers are noisy, and if the fix is wrong you have burned allocation budget and still do not know why.

The better approach is to find or construct the smallest possible reproducer. Once the small case reproduces reliably, you have a debuggable target. Only when that case is clean do you run at scale to confirm nothing regresses. This discipline also makes it far easier to share a reproducer with a teammate: “clone, run these two commands, watch it crash” is a much better bug report than “submit two hundred SLURM jobs and search the logs.”

Instrument the code before you reach for a debugger

Before attaching GDB or running Valgrind, the fastest thing you can do is add targeted fprintf calls to verify that your fix is actually being reached. This sounds obvious, and yet the most common source of “my patch does nothing” confusion is that the code path you edited is never entered for the input you are testing.

/* In work_queue.c, before the dereference that Valgrind flagged */
fprintf(stderr, "[DEBUG] task_min_resources: checking node %p on ready list\n", (void *)t);
if (!t) {
    fprintf(stderr, "[DEBUG] task_min_resources: null node encountered — skipping\n");
    continue;
}
fprintf(stderr, "[DEBUG] task_min_resources: node task_id=%d resources_requested=%p\n",
        t->taskid, (void *)t->resources_requested);

Write to stderr, not stdout, which may be buffered or redirected by the framework. Add prints before and after the critical section. If the “before” line appears but the “after” line does not, you found the crash site. If neither appears, the function is not being called — which means the bug is upstream of where you thought it was, and you just saved yourself an hour of reading the wrong code.

If the prints confirm your fix is being reached but the crash keeps happening, the root cause is elsewhere: probably the point where the stale node enters the list, not the point where it is dereferenced. Move the instrumentation upstream.

Gate these prints behind a compile-time flag before merging so they do not pollute production builds:

#ifdef WQ_DEBUG_TASK_RESOURCES
fprintf(stderr, "[DEBUG] task_min_resources: node %p task_id=%d\n",
        (void *)t, t->taskid);
#endif

Pass -DWQ_DEBUG_TASK_RESOURCES in CFLAGS to turn them on; a normal build leaves no noise.

Reach for GDB and Valgrind when instrumentation is not enough

When fprintf tells you where the crash is but not why, it is time to bring in a real debugger. For a crash you can reproduce locally, GDB is the first stop:

# Build with debug symbols and no optimization so variables are readable
./configure CFLAGS="-g -O0"
make -C work_queue/src
 
# Run until it crashes, then inspect the frame
gdb --args work_queue_worker -d all localhost 9123
(gdb) run
(gdb) bt                        # full backtrace at the crash point
(gdb) frame 2                   # switch to the frame inside task_min_resources
(gdb) p t                       # is the pointer null or garbage?
(gdb) p t->resources_requested  # does the struct look sane?
(gdb) watch t->taskid           # set a watchpoint to catch when this field changes

Valgrind slows execution by roughly ten to twenty times, so pair it with the scaled-down reproducer from the previous section. A handful of short tasks that trigger the ready-list traversal during a catalog tick is a much better target than a full production workflow.

One practical note: you generally cannot run GDB interactively inside a SLURM job. If the bug only appears at scale, use Work Queue’s -d all flag to write a verbose trace to a shared filesystem path that both the execute node and your login node can read:

work_queue_worker -d all -o /cephfs/group/myproject/wq-logs/worker-$(date +%Y%m%d%H%M%S).log \
  localhost 9123

Then reconstruct a local reproducer from what the trace tells you, and use GDB on that.

Read the logs that are already there

Before adding instrumentation or firing up a debugger, check what Work Queue already records. The -d all flag produces a structured trace across subsystems — scheduling decisions, catalog updates, task state transitions — that often answers the question before you write a single fprintf.

# Watch the debug stream in real time on a local test run
work_queue_worker -d all localhost 9123 2>&1 | grep -i "task_min\|ready\|segfault\|null\|error"
 
# After a cluster run, search a saved log
grep -n "task_min_resources\|ready_list" worker-20260609.log | head -40

The log format uses timestamps and subsystem tags, so you can narrow a twenty-thousand-line trace to the thirty lines around the catalog tick event without much effort. For the #4415 bug in particular, looking for the catalog update message immediately before the crash narrows the search window considerably.

Turn on every debug flag and log everything

When you are stuck, err heavily on the side of logging too much rather than too little. A verbose log that you have to grep through is far better than a quiet one that leaves you guessing. The more you record, the more likely it is that the crash site, the bad pointer, and the sequence of events leading to it are all sitting in the file waiting for you.

CCTools exposes this through the -d flag, which accepts one or more subsystem names. The relevant ones for a Work Queue + Makeflow investigation are wq (Work Queue task scheduling and worker communication), batch (the batch system layer that submits SLURM or Condor jobs), rmon (the resource monitor that measures cores, memory, and disk per task), and makeflow (the Makeflow DAG engine, which covers parsing, lexing, and the run loop). Pass them all:

# Worker: turn on every relevant subsystem and save to a named file
work_queue_worker -d all -o worker.log localhost 9123 &
 
# Makeflow manager: same idea, log to a separate file
makeflow -T wq -d wq,batch,rmon,makeflow -o makeflow.log mini.makeflow

-d all is the blunt instrument — it sets every bit in the flag word and includes subsystems you probably do not care about on this run. That is fine. The extra volume costs you nothing except disk space, and the subsystem tags in the log make it easy to filter later. If the log grows large enough to be unwieldy, narrow it down once you know which subsystem to focus on; start with all and restrict from there.

On the cluster, where you cannot read stderr interactively, always pair -d all with -o pointing at a shared filesystem path that your login node can reach after the job finishes. Give each job a unique filename so concurrent workers do not overwrite each other:

work_queue_worker -d all \
  -o /cephfs/group/myproject/wq-logs/worker-$(date +%Y%m%d%H%M%S)-$$.log \
  localhost 9123

The $$ expands to the worker process id on the execute node, which is a cheap uniquifier on top of the timestamp. Once the logs land, search them as a unit:

# Find the catalog tick and the crash window across all workers in a session
grep -h "task_min_resources\|catalog\|ready_list\|SIGSEGV" \
  /cephfs/group/myproject/wq-logs/worker-20260609-*.log \
  | sort -k1,2 | less

The -h flag suppresses the filename prefix so the timestamps sort cleanly. Sorting by the first two fields — date and time — stitches the per-worker logs into a single chronological view of what the whole pool was doing in the seconds before the crash. That cross-worker timeline is often what finally makes the bug obvious.

One warning: do not turn on --debug-workers in work_queue_factory and also pass -d all through --extra-options at the same time. The factory-side flag already appends -d all -o worker.<n>.log to every submission; doubling up means every worker writes two interleaved logs to slightly different paths and you end up reading redundant output. Pick one approach per session.

The quick checklist before every cluster run

Before you fire off a batch job, spend sixty seconds on these:

• Did you rebuild and reinstall after your last edit? A stale binary is the most common source of “my fix does nothing.”
• Is the debug log going to a path that exists and is writable from the execute node?
• Is the job small enough that you will get a result in under ten minutes, or do you have a good reason to go bigger right now?
• Did you commit your current state so that whatever happens on the cluster, the code is recoverable?
• Did you tell a teammate what you are about to test, so they can flag it if they know something relevant?

None of these take long. Together they prevent the most common ways a cluster debugging session turns into a wasted afternoon.

Last week, I had the opportunity to attend Throughput Computing Week 2026 in Madison, Wisconsin, organized by the Center for High Throughput Computing (CHTC). I participated in the conference on Wednesday and Thursday and came away with a deeper appreciation for the diverse ways high-throughput computing is impacting research across disciplines.

One of the highlights of the event was attending my advisor Dr. Douglas Thain’s talk on Thursday, “Wrangling Massive Task Graphs with VineReduce.” The presentation showcased new approaches for managing large-scale computational workflows and demonstrated the continued evolution of distributed computing systems.

Beyond the technical sessions, one of the most rewarding aspects of the conference was meeting researchers from a wide range of scientific domains. I spoke with teams working in space research who use HTCondor to process massive amounts of satellite-generated terrain data. Other researchers shared how high-throughput computing is accelerating genome analysis in biology and enabling large-scale weather modeling that helps civil engineers design more efficient roads and buildings.

Among the many talks, I was particularly fascinated by discussions around Pelican and its role in distributed data storage and access. I also learned about emerging challenges surrounding next-generation hardware accelerators and AI-focused chips. Researchers and computing teams from universities across the country discussed how they are balancing growing computational demands through cloud resources, strategic infrastructure investments, and cost optimization techniques.

As artificial intelligence continues to reshape science and engineering, the conference highlighted how computing infrastructures are evolving alongside it. From data-intensive scientific discovery to large-scale AI workloads, the conversations at Throughput Computing Week reinforced how rapidly the world of research computing is changing—and how important collaboration across disciplines will be in navigating that future.

This event brought together experts from 10 different experiments to align on shared challenges and collaborate on community solutions. Beyond establishing a common technical language, we mapped out key focus areas for the future:

• Resource optimization: Improving execution behavior and utilization.
• Data-aware late binding: Navigating data locality and streaming.
• Request, provenance & validation: Exploring unified standards for workflow interfaces.
• AI in operations: Leveraging AI assistance in complex distributed systems.

The conversation is just getting started. If you’re interested in the evolution of Workflow/Workload Management and community standards, we’d love for you to join us! Connect via our Mattermost channel

This May CCL third-year PhD student Jin Zhou traveled to New Orleans for IPDPS 2026, the 40th IEEE International Parallel & Distributed Processing Symposium, held at the Marriott on Canal Street from May 25 to 29. He presented our paper SciWIND: Effectively Exploiting Node-Local Storage for Data-Intensive High-Energy Physics Workflows, which looks at how to use node-local scratch more deliberately when large HEP workflows run on opportunistic clusters and workers fail mid-run. The talk was a nice cap on a line of work the lab has been pushing through TaskVine and our HEP collaborations, and it was good to put the system in front of people who live with scheduling, storage, and workflow engines every day.

Between sessions Jin followed the conference program: tutorials and workshops on the first two days, then the main track, keynotes, and plenty of hallway conversations. IPDPS still has that familiar mix of parallel algorithms, distributed systems, and applications at scale. AI was clearly a hot topic this year, showing up in keynotes, panels, and hallway chats about training and inference at scale. A few questions after the SciWIND talk turned into longer conversations about eviction recovery, disk pressure on shared filesystems, and where workflow runtimes should own policy versus leave it to the user. New Orleans helped too, with late walks along Canal Street and coffee between sessions that made the week feel less like a sprint and more like a real meeting of the community.

Thanks to everyone who came to the SciWIND session and to the IPDPS organizers for another smooth run. We are glad the paper is out in the proceedings and happy to keep the conversation going with groups wrestling with the same storage and resilience headaches in production science workflows.

This week our lab was in Chicago for GCASR 2026—the Greater Chicago Area Systems Research Workshop (gcasr.org), a regional gathering focused on computer systems, from operating systems and distributed infrastructure to the tools that make large-scale science tractable. It is a friendly venue for students and faculty to share work in progress, compare notes on real systems problems, and meet peers from departments and labs around the Midwest.

The CCL makes this trip every year: Chicago is home base for GCASR, and showing up has become part of how we stay plugged into the systems community between the bigger conference cycles.

Each student on the team presented a research poster, as we typically do at this venue. The poster floor was busy, with good questions, fast feedback, and plenty of hallway conversations that do not fit on a slide deck. Between sessions, people attended invited talks and panels, swapped implementation details, and followed threads from scheduling and storage to workflows and AI-facing infrastructure.

Thanks to everyone who stopped by our posters and to the organizers for another well-run GCASR. We appreciated the insights, the introductions, and the chance to catch up with colleagues in person. We look forward to seeing familiar faces at GCASR 2027!

Once you are hacking vine_worker in C and feeding workers through HTCondor, two boring problems show up over and over: the pool still runs yesterday’s binary, and your printfs never appear next to the Python manager. This note is about staging the binary you meant to ship and getting worker-side trace somewhere useful. For the repo layout start with A Beginner’s Map to the CCTools Codebase; for the usual vine_submit_workers / vine_factory workflow see Submitting Workers to a Cluster.

None of that changes the basic fact that a live worker runs whatever was staged when the job went in, not necessarily whatever make install just dropped into $PATH. To pick up a new build you still have to drain the old worker jobs, stop a factory that is holding a stale scratch copy if one is running, install, and submit again.

The install that actually reaches HTCondor

After ./configure, we still do the blunt rebuild from taskvine/src when we want zero ambiguity:

make clean && make && make install

On your own machine you probably have a vine_worker running already. Stop it before make clean && make && make install. Overwriting the on-disk binary while the process still has it mapped is a good way to take a nose dive, sometimes straight into a segfault, and whatever tasks it was holding will go with it. Bring the worker back up against the manager after the install.

On the cluster there is another wrinkle. vine_submit_workers drops a copy of vine_worker into its submit directory when you run it; vine_factory copies whatever vine_worker resolves to (unless you pointed it at --worker-binary) into -S,--scratch-dir once at startup, and every batch job runs ./vine_worker out of that directory. Tweaking $PATH later while the factory is still running does nothing for jobs already in flight. When you need a clean slate, condor_rm tears down old worker jobs (plenty of sites use condor_rm -all to wipe your queue); kill the factory on the submit side if it is still recycling the old scratch tree; install; submit again so what landed in the staging directory matches what you think you built.

Turning on logs people can open

Extra fprintf lines in vine_worker are not going to show up in your Python manager transcript; you still need worker-side debug. -d all is the heavy-handed default that turns the noise up across subsystems. Add -o with a path when stderr is useless because you will never SSH to that execute node (CLI reference).

Sanity check on the laptop first:

vine_worker -d all localhost 9123

You want a wall of debug text right away; same flags later ride along inside Condor.

vine_factory --debug-workers appends -d all -o worker.<n>.log for each submission (the number is a factory-side counter, not Condor’s proc id). Those files hop back in the job sandbox when the run finishes because Condor uses when_to_transfer_output = on_exit, which is fine for an autopsy and useless if you wanted tail -f while the slot was still busy.

For that you need a directory the execute node and your login node both mount. Hand the worker -d all and -o /absolute/path/... through vine_factory -E,--extra-options. Under the hood the Condor batch module writes a tiny condor.sh wrapper that does eval "$@" with the full command line as arguments (batch_queue_condor.c).

Put something unique to each job in the filename. HTCondor’s $(Cluster) and $(Process) (or the $(ClusterId) / $(ProcId) spellings your site prefers) expand per proc; write them as \$(Cluster) and \$(Process) when an earlier shell would swallow the $ before Condor sees them.

Choose a parent dir d on the shared filesystem, maybe stamp ts when you kick off a factory, maybe add something like ipid if your wrapper exports it before vine_worker starts. The filename pattern is your own business as long as two workers never write the same path by accident.

d=/cephfs/group/myproject/vine-logs
ts=$(date +%Y%m%d%H%M%S)
vine_factory -T condor -M myproject -S /scratch/$USER/vf \
  -E "-d all -o ${d}/worker-${ts}-\$(Cluster).\$(Process).log"

ts in that example is just a label for one factory session; \$(Cluster).\$(Process) keeps different Condor procs from stepping on each other. If you already have something like ipid in the environment and you trust it to be unique per writer, ${d}/worker-${ts}-${ipid}-\$(Cluster).\$(Process).log is the same idea with more belt and suspenders.

Do not turn on --debug-workers and a fat custom -E at the same time unless you like reading everything twice.

From Python the same string can ride through vine.Factory.extra_options, which maps to vine_factory --extra-options. There is still no debug_workers knob in the binding, so either stay explicit in extra_options or shell out to the CLI if you need the factory-only switch:

import ndcctools.taskvine as vine

manager = vine.Manager(9123)
factory = vine.Factory(batch_type="condor", manager=manager, log_file="factory.log")
factory.extra_options = r"-d all -o /cephfs/group/myproject/vine-logs/worker-\$(Cluster).\$(Process).log"
with factory:
    job = vine.Task("echo heartbeat")
    manager.submit(job)
    manager.wait(timeout=120)

When you pack up, skim the usual checklist: did the slot actually run the binary you staged, is -d all going somewhere you can read, and did you give each job a distinct path (Condor macros plus any label you added on purpose) so the logs never clobber each other.

A TaskVine application becomes useful on a cluster only after workers are placed where the compute capacity is. On a laptop, we may start a worker by hand:

vine_worker HOST PORT

That is fine for a local test. It is not how we want to run a real workflow on a shared system. A cluster already has a resource manager, and TaskVine should use it. CCTools supports worker submission through several batch systems, including HTCondor, Slurm, and UGE. At the Notre Dame CRC compute cluster, the common path for our TaskVine runs is HTCondor, so this post uses Condor as the concrete example.

There are two ways to think about cluster workers. vine_submit_workers is the direct path. It submits a fixed number of workers and then gets out of the way. vine_factory is the managed path. It keeps watching the manager, estimates how many workers are needed, submits new workers when old ones disappear, and writes logs that are much easier to use during a postmortem.

The TaskVine User’s Manual covers both tools. Here we focus on the HTCondor path and the Conda environment details that tend to matter in real runs.

The runtime shape

A TaskVine manager listens on a host and port, or advertises itself under a project name through the catalog. Workers connect back to that manager. The batch system does not run your manager. It runs worker jobs that eventually execute vine_worker with the right connection arguments.

That distinction explains the first command most users see in the manual:

vine_submit_workers -T condor MACHINENAME 9123 10

This asks HTCondor to start ten workers. When those jobs begin to run, each worker calls back to MACHINENAME:9123. If your manager uses a project name, the command becomes:

vine_submit_workers -T condor -M myproject 10

The project name form is usually easier once runs become repeatable, because workers can discover the manager from the catalog instead of hard coding a hostname and port.

Direct submission with vine_submit_workers

vine_submit_workers is a shell script in CCTools. It supports Condor, Slurm, and UGE through -T or --batch-type. The Condor path is useful to understand because it shows what the tool is actually doing for us.

For Condor, the script builds a small submission directory, usually /tmp/$USER-workers. It copies vine_worker into that directory. It also copies helper inputs such as cctools_gpu_autodetect. If --poncho-env is used, it also copies poncho_package_run and the packaged environment. Then it writes a worker.sh wrapper and a condor_submit_file.

The generated Condor submit file uses worker.sh as the executable. It transfers the worker binary and all declared input files. It writes output to files named like worker.$(CLUSTER).$(PROCESS).output, error output to worker.$(CLUSTER).$(PROCESS).error, and Condor job events to workers.log. It also sets getenv = true, so the submitted job inherits the submit side environment unless you change the generated file or Condor policy rejects it.

That automation is the main convenience. A practical HTCondor command looks like this:

vine_submit_workers \
  -T condor \
  --cores 4 \
  --memory 8192 \
  --disk 20000 \
  --poncho-env dv5-env.tar.gz \
  -M myproject \
  20

This submits twenty workers. Each worker advertises four cores, 8192 MB of memory, and 20000 MB of disk to TaskVine, and the Condor submit file requests matching resources from Condor.

One common gotcha is argument order. The script parses options until it reaches the final positional arguments. Put the manager address, or the worker count for a project name, at the end. Also put -T condor before Condor specific options such as --requirements, because the script only knows how to parse those options after the batch system has been selected. The two accepted shapes are:

vine_submit_workers [options] HOST PORT NUM_WORKERS
vine_submit_workers [options] -M PROJECT NUM_WORKERS

Another easy mistake is localhost. The script explicitly rejects localhost for Condor workers. A worker running on a remote execute node would interpret localhost as the execute node itself, not the manager. Use a routable hostname or a project name.

Finally, vine_submit_workers submits a fixed number of jobs and exits. In the Condor path, the script writes queue NUM_WORKERS, runs condor_submit, and does not leave behind a monitor. If a worker exits because of a worker side bug, a segmentation fault, preemption, a node problem, or a random distributed systems failure, the submit command is no longer around to replace it. The manager can retry interrupted tasks when another worker is available, but the worker pool itself may silently shrink. You need to notice that through condor_q, TaskVine status, manager logs, or slow progress, and then resubmit workers yourself.

Common vine_submit_workers options

Do not treat vine_submit_workers --help as a checklist. Most runs only need a small part of it.

Where to submit:

• -T, --batch-type condor|slurm|uge: choose the batch system. On Notre Dame CRC, we usually use condor.
• HOST PORT NUM_WORKERS: connect workers directly to a manager address.
• -M, --manager-name PROJECT NUM_WORKERS: connect workers to a manager advertised through the catalog.

How many resources each worker should have:

• --cores N: request this many cores and report the same number to TaskVine.
• --memory MB: request this much memory and report it to TaskVine.
• --disk MB: request this much disk and report it to TaskVine.

How workers run:

• -t, --timeout SECONDS: let idle workers exit after this time. The default is 900 seconds.
• --scratch-dir PATH: choose where the local submit directory is created.
• --poncho-env FILE.tar.gz: run the worker inside a packaged Poncho environment.
• --dry-run: print the generated worker script and Condor submit file instead of submitting. Use this before submitting if the command has changed.

Condor placement:

• -r, --requirements EXPR: add a Condor requirements expression.

The remaining flags are for less common site policies and advanced worker behavior. Leave them out of the first version of a run. When one of those cases comes up, check the TaskVine manual and the vine_submit_workers source.

Why a factory is usually better

Direct submission is convenient when the run is small, short, or interactive. For a serious workflow, the better default is vine_factory.

The factory is a long running process. It periodically asks the manager, or the catalog, what work exists and how many workers are connected. It then compares the desired worker count with the workers it has already submitted. If it needs more workers, it submits them through the selected batch system. If workers exit, it observes those exits, removes them from its internal job table, and can submit replacements on the next cycle.

That behavior matters on shared clusters. HTCondor can preempt jobs. Nodes can fail. Worker processes can crash. A manager can keep retrying tasks, but it cannot create new batch jobs by itself. The factory fills that gap.

A normal Condor factory command looks like this:

vine_factory \
  -T condor \
  --min-workers 2 \
  --max-workers 50 \
  --workers-per-cycle 10 \
  --cores 4 \
  --memory 8192 \
  --disk 20000 \
  --poncho-env dv5-env.tar.gz \
  --manager-name myproject

This keeps at least two workers and at most fifty workers available for managers named myproject. The factory checks the manager periodically, estimates worker demand from waiting and running tasks, caps the target at --max-workers, raises it to --min-workers if needed, and submits at most --workers-per-cycle new jobs in a single cycle.

The implementation keeps an internal table of submitted batch job IDs. Each submitted worker runs a generated vine_worker command with the manager target, timeout, and requested resource shape. The factory then calls the batch queue layer to submit that job. Later, it waits for completed jobs with a short timeout. When a known worker job exits, the factory decrements its submitted count. That is what lets it replace lost workers without you watching condor_q all afternoon.

The logs are also useful. vine_factory -d vine -o factory.log gives you a persistent record of what the factory thought the manager needed, how many jobs it submitted, and which jobs exited. --debug-workers adds per worker debug logs in the factory scratch directory. When a run fails at scale, this is much easier to debug than a one time submit command whose only durable state is in Condor.

Common vine_factory options

The factory has many knobs because it is a controller, not only a submit command. For normal Condor use, start with these.

Where to submit and which manager to serve:

• -T, --batch-type TYPE: required batch system type. Use condor for HTCondor.
• -M, -N, --manager-name PROJECT: serve managers matching a project name or regular expression.
• HOST PORT: connect directly to one manager instead of using the catalog.
• -C, --config-file FILE: read options from a JSON config file. The factory re-reads this file periodically.

How many workers to keep around:

• -w, --min-workers N: keep at least this many workers.
• -W, --max-workers N: never ask for more than this many workers.
• --workers-per-cycle N: limit how many new workers the factory submits in one cycle.
• -t, --timeout SECONDS: idle timeout passed to each worker.
• --factory-period SECONDS: how often the factory re-evaluates demand.

Worker resources:

• --cores N: request and advertise cores per worker.
• --memory MB: request and advertise memory per worker.
• --disk MB: request and advertise disk per worker.
• --gpus N: request and advertise GPUs per worker.

Worker environment and logs:

• -S, --scratch-dir PATH: choose the factory scratch directory.
• -d, --debug SUBSYSTEM: enable factory debugging.
• -o, --debug-file FILE: write factory debug output to a file.
• --debug-workers: create worker side debug logs.
• --poncho-env FILE.tar.gz: run each worker in a Poncho package.

Condor placement:

• --condor-requirements EXPR: add Condor requirements. Multiple uses are combined.

The remaining factory options are for less common site policies and advanced worker setups. Leave them out of the first version of a run unless your cluster setup requires them. For the complete set, use the TaskVine manual and the vine_factory source.

A config file is often cleaner than a long command:

{
  "manager-name": "myproject",
  "min-workers": 2,
  "max-workers": 50,
  "workers-per-cycle": 10,
  "factory-period": 30,
  "cores": 4,
  "memory": 8192,
  "disk": 20000,
  "condor-requirements": "TARGET.OpSysAndVer == \"AlmaLinux9\""
}

Then start the factory with:

vine_factory -T condor -C factory.json --poncho-env dv5-env.tar.gz

The config file is re-read while the factory runs, so you can raise max-workers, lower min-workers, or adjust resource settings without restarting the controller.

A small wrapper for regular CRC runs

After a command becomes part of a weekly workflow, it is worth wrapping the repeated parts. In our day to day runs, we usually know the manager project name, the desired number of workers, the per worker resource shape, the scratch location, and the Poncho package. A thin shell wrapper makes that intent easier to read than a long vine_factory command.

Here is a cleaned up version of the pattern:

#!/usr/bin/env bash

MANAGER_NAME=dagvine-manager
N_WORKERS=40
CORES=16
MEMORY_GB=12
DISK_GB=100
PONCHO_ENV=dagvine-env.tar.gz
SCRATCH_DIR=/scratch365/$USER/factory_dv5
CONDOR_REQUIREMENTS='((has_vast))'

while [ $# -gt 0 ]
do
    if [ "$1" = "-M" ] || [ "$1" = "--manager-name" ]
    then
        MANAGER_NAME="$2"
        shift 2
    elif [ "$1" = "--workers" ]
    then
        N_WORKERS="$2"
        shift 2
    elif [ "$1" = "--cores" ]
    then
        CORES="$2"
        shift 2
    elif [ "$1" = "--memory" ]
    then
        MEMORY_GB="$2"
        shift 2
    elif [ "$1" = "--disk" ]
    then
        DISK_GB="$2"
        shift 2
    elif [ "$1" = "--poncho-env" ]
    then
        PONCHO_ENV="$2"
        shift 2
    else
        echo "unknown option: $1" >&2
        exit 2
    fi
done

MEMORY_MB=$((MEMORY_GB * 1024))
DISK_MB=$((DISK_GB * 1024))

vine_factory \
    -T condor \
    --scratch-dir "$SCRATCH_DIR" \
    --poncho-env "$PONCHO_ENV" \
    --condor-requirements "$CONDOR_REQUIREMENTS" \
    --manager-name "$MANAGER_NAME" \
    --min-workers "$N_WORKERS" \
    --max-workers "$N_WORKERS" \
    --workers-per-cycle "$N_WORKERS" \
    --cores "$CORES" \
    --memory "$MEMORY_MB" \
    --disk "$DISK_MB" \
    --timeout 36000

The wrapper intentionally sets min-workers, max-workers, and workers-per-cycle to the same value. That makes the command behave like a fixed size pool, but with the factory still watching for worker exits and replacing lost jobs. The --timeout 36000 setting keeps workers around for long runs. The --condor-requirements '((has_vast))' line is site specific. Keep it if your workflow needs that Condor attribute, and replace it with the requirement expression that matches your cluster policy otherwise.

The script also accepts resource values in GB for memory and disk, then converts them to MB because vine_factory --memory and --disk expect megabytes. That small conversion removes a common source of mistakes when we are switching between Condor habits, shell scripts, and TaskVine options.

Conda environments and Poncho

Mismatched Conda environments create failures that look unrelated to worker submission. The worker binary, Python packages, shared libraries, and task imports should agree with the manager side assumptions. Start the manager, vine_submit_workers, and vine_factory from the same intended Conda environment whenever possible.

For Python heavy workflows, package that environment and attach it to every worker. The workflow is small:

conda activate dv5-env
poncho_package_create "${CONDA_PREFIX}" dv5-env.tar.gz

Then pass the tarball to the submitter:

vine_submit_workers -T condor --poncho-env dv5-env.tar.gz -M myproject 20

or to the factory:

vine_factory -T condor --poncho-env dv5-env.tar.gz --min-workers 2 --max-workers 50 -M myproject

The details are handled for you. vine_submit_workers ships poncho_package_run and the tarball as Condor input files, then runs ./poncho_package_run -e ENV.tar.gz -- ./vine_worker .... vine_factory implements --poncho-env as a wrapper around the worker command and adds both the wrapper tool and package as worker inputs. In both cases, the worker starts inside the packaged environment before it connects to the manager.

HTCondor operations we actually use

Once workers are submitted, Condor is still the place to inspect and clean up batch jobs.

Check your queued and running worker jobs:

condor_q

Remove workers when you want to stop cleanly:

condor_rm -all

On a shared system, be careful with -all. It removes your Condor jobs, and that is normally what we want when a TaskVine experiment needs a clean restart. After removal, also check for leftover local processes on the submit host:

ps aux | grep vine_worker
ps aux | grep vine_factory

This matters because a stale factory can quietly submit new workers after you think the old run is gone. A stale worker or wrapper process can also leave the environment in a state that does not match the next manager run. Clean up before changing Conda environments, project names, worker resources, or package tarballs.

Condor also lets you enter a worker sandbox:

condor_ssh_to_job JOBID

This is useful when a worker behaves differently on the execute node than it does on the submit node. You can inspect transferred files, the unpacked Poncho environment, logs, temporary outputs, and the actual command environment. For a worker crash, combine condor_ssh_to_job, worker.$CLUSTER.$PROCESS.error, factory logs, and manager logs before changing code.

Which tool should you use

Use vine_submit_workers at the beginning. It is a good way to run a very small workflow, or a simple bag of tasks, and check that the basics are correct: the manager is reachable, workers can start on Condor, the Poncho package works, and the requested resources make sense.

After that first check, switch to vine_factory and make it the habit. Normal cluster work should not depend on manually noticing that workers disappeared. The factory has a control loop. It notices exited worker jobs, keeps the pool between min-workers and max-workers, throttles new submissions with workers-per-cycle, adapts to manager demand, and leaves behind logs that explain its decisions.

The manager still owns task scheduling and retries. Condor still owns placement and preemption. The factory sits between them and keeps the worker supply healthy. A good workflow is to validate with vine_submit_workers, then run real workloads with vine_factory.

Colin is a Ph.D. student in Computer Science and Engineering at Notre Dame and works in our group with Prof. Douglas Thain. His research interests are distributed systems and scientific workflows on clusters. Much of his work so far has centered on Parsl and TaskVine, on how scheduling and data placement behave when workflows mix a parallel filesystem with node-local disks, and on keeping related tasks and their intermediate files together instead of shuffling everything through shared storage. With candidacy behind him, we are looking forward to the next stretch of the Ph.D. and what he will build and publish from here.

Nice work on candidacy, Colin.