You submit a batch job, wait in the queue, and suddenly see its status transition to FAILED or HOLD. When you open your output directory, you are greeted by a 200-line wall of obscure error messages, memory dumps, and cryptic exit codes.

It is a classic high-performance computing friction point: feeling overwhelmed by stack traces and assuming something catastrophic went wrong. But HPC error logs are structured breadcrumbs, not punishments. Once you know where your scheduler writes these files and how to parse the output, identifying the root cause takes seconds across any Workload Manager—whether your cluster uses SLURM, Univa Grid Engine (UGE / SGE), or HTCondor.

Standard Output vs. Standard Error vs. Event Logs

When running non-interactive batch jobs, system output streams split into separate files depending on your workload manager:

• Standard Output (.out / stdout): Reserved for standard runtime logs, progress bars, print() statements, and calculation results.
• Standard Error (.err / stderr): Reserved for system warnings, thrown exceptions, segmentation faults, and library errors.
• Workflow Event Log (.log): Unique to schedulers like HTCondor, this file records lifecycle events managed by the scheduler itself (e.g., job submission, node assignment, resource tracking, and hold reasons).

Here is how you define these log files across the three major schedulers:

SLURM (.sub / .sh)

#SBATCH --output=my_job_%j.out
#SBATCH --error=my_job_%j.err

(Where %j inserts the unique SLURM Job ID).

Univa Grid Engine / SGE (.sub / .sh)

#$ -o my_job_$JOB_ID.out
#$ -e my_job_$JOB_ID.err

(Where $JOB_ID inserts the unique Grid Engine Job ID).

HTCondor (.submit)

output = my_job_$(Cluster).out
error  = my_job_$(Cluster).err
log    = my_job_$(Cluster).log

(Where $(Cluster) represents the HTCondor Cluster/Job ID).

When a job fails, always inspect the .err file first (and check the .log file if you are using HTCondor).

Reading Traces: Top-to-Bottom vs. Bottom-to-Top

Reading an error log sequentially from line 1 to line 200 is often a waste of time. Different programming languages and runtime tools print stack traces in entirely different directions.

Python: Read Bottom-to-Top

Python prints error traces chronologically, meaning the actual fatal exception is always printed at the very bottom of the trace.

• The Strategy: Scroll directly to the final line of the .err file to identify the exact error type (e.g., KeyError, IndexError, ModuleNotFoundError). Once you know what failed, scan upward to find the line pointing to your script rather than an internal library package.

C/C++ and Fortran: Read Top-to-Bottom

Compiled languages and build tools log errors chronologically as they are encountered during compilation or execution.

• The Strategy: Start at the very top of the error log. The first error printed is almost always the true root cause; the hundreds of lines following it are usually cascading failures triggered by that initial crash.

Bash / Shell Scripts: Read Where It Stops

By default, Bash scripts continue executing subsequent commands even if an earlier line fails (unless set -e is enabled).

• The Strategy: Search for the specific command line that failed by scanning upward from where output stopped matching your expected workflow.

Decoding the Usual Suspects: Classic HPC Failures

Most cluster job crashes stem from three recurring issues. Recognizing how each scheduler reports these failures allows you to fix them instantly.

1. Out Of Memory (OOM) / Exit Code 137 / Job Holds

• The Cause: Your program tried to consume more RAM than requested. To prevent your job from crashing neighboring processes on a shared node, the system kernel or scheduler terminated your process.
• How Schedulers Report It:
• SLURM: Your .err file or sacct -j <job_id> reports exit code 137 (128 + Signal 9 SIGKILL), often accompanied by slurmstepd: error: Detected allocation failure. OOM Killer invoked.
• UGE / SGE: The job fails with exit code 137. Running qacct -j <job_id> or qstat -j <job_id> shows failed: 100 : Assumed OS problem or reveals that the job exceeded its specified maxvmem ceiling.

• HTCondor: Rather than exiting outright, HTCondor usually places the job in a HOLD state (H). Running condor_q -hold or checking the .log file explicitly states: Job used more memory than requested.

• The Fix: Increase the requested memory in your submission script:
• SLURM: #SBATCH --mem=32G (or #SBATCH --mem-per-cpu=8G)
• UGE: #$ -l h_vmem=32G
• HTCondor: request_memory = 32GB

2. “No Space Left on Device” or Quota Exceeded

• The Symptom: IOError: [Errno 28] No space left on device or write error: disk quota exceeded.
• What it means: Your script ran out of disk space. This rarely happens on large global storage filesystems (/scratch or /project), but frequently occurs when temporary files auto-write to small localized /tmp partitions or constrained home directories (/home/username).
• The Fix: Redirect temporary directory environment variables to high-capacity scratch space directly inside your job script before running code:

export TMPDIR=/scratch/username/tmp
export PIP_CACHE_DIR=/scratch/username/pip_cache
mkdir -p $TMPDIR $PIP_CACHE_DIR

(In HTCondor, you can also request dedicated local scratch disk space via request_disk = 50GB in your submit file).

3. “Command Not Found” or “ModuleNotFoundError”

• The Symptom: bash: line 14: gfortran: command not found or ModuleNotFoundError: No module named 'torch'.
• What it means: Compute nodes start in a clean, isolated environment. They do not automatically inherit loaded modules, exported $PATH variables, or active Conda environments from your login session terminal.
• The Fix: Explicitly load system modules and activate virtual environments directly inside your execution script:

# Load system modules
module load python/3.11

# Activate your custom environment
source /scratch/username/envs/my_env/bin/activate

# Execute your application
python3 my_script.py

Quick Scheduler Diagnostic Cheat Sheet

When a job finishes or fails unexpectedly, use these diagnostic commands to query job metadata and exit status:

The Log Parsing Checklist

Before reaching out to cluster administrators or teammates for help with a broken job, run through these steps:

1. Locate the standard error log (.err). For HTCondor, inspect the workflow .log file as well for scheduler-level events.
2. Determine the language direction. Read Python errors from the bottom up; read C/C++ build failures or shell errors from the top down.
3. Use scheduler diagnostics. Check sacct (SLURM), qacct (UGE), or condor_q -hold (HTCondor) to verify exit codes and memory usage.
4. Identify resource limits. An exit status of 137 or a memory hold reason means your job needs a higher memory allocation.
5. Set explicit pathing. Ensure all module load commands, TMPDIR paths, and environment activations are written explicitly inside your job submission file.

Developing code locally inside a Jupyter Notebook or stepping through lines of Python with VS Code’s debugger is a fast, intuitive way to work. But when your local machine runs out of RAM, CPU cores, or GPU memory, you pack up your workflow and move to a High-Performance Computing (HPC) cluster.

Once there, many researchers hit an immediate friction point: clusters are traditionally built around non-interactive batch scripts submitted to job schedulers like SLURM or Grid Engine. Switching from an interactive graphical editor back to pure command-line scripts and print-statement debugging can feel like stepping backward in time.

Fortunately, you don’t have to give up your interactive tools to use a supercomputer. You just need to know how to bridge local GUIs with remote cluster compute—without breaking cluster rules.

The Cardinal Sin: Running Servers on Shared Login Nodes

When you SSH into a cluster, you land on a login node (or head node). Think of the login node as a shared front porch. It is meant exclusively for light administrative tasks: editing text files, organizing directory trees, checking queue status, and submitting job scripts.

The most common mistake new cluster users make is running jupyter notebook or connecting VS Code’s Remote-SSH plugin directly to the login node.

Why this breaks the cluster:

• VS Code Server (.vscode-server) spins up background Node.js processes, language servers, file watchers, and extension host daemons that constantly index workspace directories.
• Jupyter Notebooks execute heavy data manipulation, model training, and plotting live inside the server process.

Login nodes are shared simultaneously by hundreds of users. When an interactive server process starts hogging gigabytes of RAM and maxing out CPU cores, the entire command-line interface bogs down for everyone else. To prevent system crashes, administrators install automated automated watcher scripts that will abruptly kill any long-running or high-memory user process on a login node.

To run these tools properly, your server processes must run on a compute node.

Strategy 1: SSH Port Forwarding (ssh -L)

If you want to run Jupyter or VS Code manually, you can use SSH Local Port Forwarding (ssh -L). This technique creates a secure tunnel between a port on your local laptop, through the login node, and directly into an active compute node.

Step 1: Request an Interactive Compute Session

Before launching any server, request a temporary allocation on a compute node using your cluster’s scheduler:

# Request 2 CPU cores, 16GB RAM, and 2 hours on SLURM
srun --tasks=1 --cpus-per-task=2 --mem=16G --time=02:00:00 --pty bash

Once your shell transfers you to a compute node, take note of the node’s hostname (e.g., node042.cluster.internal).

Step 2: Launch the Server on the Compute Node

Start your Jupyter server on the compute node, making sure to disable automatic browser opening:

# Load your Python module or activate your Conda environment first
module load python
jupyter notebook --no-browser --port=8888

Terminal output will provide a URL containing a security token (e.g., http://localhost:8888/?token=a1b2c3d4...).

Step 3: Establish the SSH Tunnel from Your Local Laptop

Open a new terminal tab on your local laptop (not on the cluster) and set up the tunnel using ssh -L:

# Syntax: ssh -L [Local_Port]:[Compute_Node_Hostname]:[Remote_Port] [Username]@[Login_Node]
ssh -L 8888:node042.cluster.internal:8888 netid@cluster.university.edu

This command forwards traffic from your laptop’s local localhost:8888 through the login node jump host directly to port 8888 on compute node042.

Step 4: Access the Interface

Open your local web browser and navigate to http://localhost:8888. Paste the security token generated in Step 2, and you now have a full Jupyter GUI running on dedicated cluster hardware!

VS Code Remote Tip: When using VS Code’s Remote SSH extension on shared systems, home directory file locking can sometimes break connections. Refer to resources like Notre Dame’s CRC VS Code Documentation for cluster-specific settings, such as enabling “Lockfiles In Tmp” in your extension settings to ensure seamless reconnects.

Strategy 2: Open OnDemand (The Modern Web Portal)

Setting up SSH tunnels manually works well, but it requires managing port numbers, tokens, and multiple terminal windows. Many modern HPC centers simplify this entirely by deploying Open OnDemand (OOD).

Open OnDemand is an open-source, web-based portal that gives you graphical access to cluster resources straight through a web browser without needing any local SSH configuration or command-line tunneling.

How Open OnDemand Works:

1. Log in via Web Browser: Navigate to your university’s Open OnDemand URL (e.g., ondemand.crc.nd.edu) and authenticate with your institutional login.
2. Select an Interactive App: Choose Jupyter, VS Code, RStudio, or MATLAB from the interactive applications menu.
3. Configure Your Resource Allocation: A form will ask you to specify your required resources (number of CPUs, GPUs, RAM amount, and maximum runtime).
4. Launch: Click Launch.

Under the hood, Open OnDemand automatically writes a job script, submits it to the cluster scheduler (SLURM/Grid Engine), waits for a compute node to be allocated, launches the server process, builds a secure web proxy, and presents a “Connect” button in your browser window.

Because Open OnDemand routes all compute through scheduled jobs, you get the full speed of dedicated compute hardware and GPUs without violating cluster usage policies or risking process kills on login nodes.

Summary: The Interactive Computing Checklist

Interactive development and HPC compute can easily coexist if you follow these rules:

1. Never run Jupyter Notebooks or VS Code Server processes on login nodes. Keep login nodes clean for light editing and job submission.
2. Check for Open OnDemand first. If your site hosts an Open OnDemand portal, use it for one-click interactive web sessions.
3. Allocate compute nodes before launching servers manually. Always start with an interactive job allocation (srun or qsub).
4. Tunnel securely with ssh -L. Forward local ports through the login host to the specific compute node hosting your active server.
5. Clean up when finished. Terminate your interactive compute sessions and close notebook servers when you finish working so cluster resources return to the queue for other researchers.

This July, CCL lab member Lax traveled to Montreal, Canada for FSE 2026, the ACM International Conference on the Foundations of Software Engineering, to present our paper SADE-SIM: A Scalable Simulation Platform for Validating City-Scale Multi-sUAS Missions.

SADE-SIM is a hybrid simulation framework that pairs physical accuracy with photorealistic visualization to support large multi-sUAS deployments at city scale. It’s built to close a gap in existing tools, which typically handle only a handful of concurrent vehicles and fall short on the fidelity needed to evaluate vision-based navigation, sensor degradation in urban environments, and compliance with evolving airspace regulations. The platform brings together web-based mission planning, real-time control compatible with standard protocols, live video stream visualization, and archival review of flight data.

To show what the system can do, Lax presented a case study built around a 32-sUAS heterogeneous fleet flying diverse mission profiles through a restricted airspace zone, where vehicles dynamically request entry and the system evaluates mission-critical safety parameters before granting access.

Between sessions, Lax followed the conference program, catching keynotes, technical tracks, and plenty of hallway conversations along the way. FSE still has that familiar, vibrant mix of software engineering foundations, automated testing, and cyber-physical systems validation. Testing and validating autonomous systems was clearly a hot topic this year, coming up again and again across sessions and informal chats about deployment safety at scale.

Thanks to everyone who came to the SADE-SIM session and to the FSE organizers for another smooth run. We’re glad to have the paper out in the proceedings, and we’re looking forward to continuing these conversations with researchers and practitioners working on the safety, validation, and scalability of autonomous aerial systems.

You can view the presentation slides here.

If you are transitioning from running code on your personal laptop to using a high-performance computing (HPC) cluster, you are about to encounter a major cultural shock: you do not have sudo privileges. On your own machine, installing a new tool is as simple as running sudo apt-get install, brew install, or clicking an installer. On a shared cluster, these commands will reward you with a blunt Permission denied. This restriction isn’t just bureaucratic cruelty; it is a fundamental safety measure. In a shared environment with hundreds of users, giving everyone root access would lead to broken configurations, security breaches, and software chaos.

So, how do you install the specific libraries, languages, and tools your research requires?

HPC systems use three primary tools for software management: Environment Modules, Conda, and Containers. Each has its own strengths, and using them correctly will save you days of troubleshooting—and keep you from accidentally drawing the ire of the cluster administrators.

Level 1: Environment Modules (The Built-In Way)

Before you try to download or compile anything yourself, you should check if the cluster administrators have already installed it for you. HPC systems use an environment management tool called Modules to manage hundreds of pre-installed, highly optimized software packages.

Because different researchers need different versions of the same software (e.g., Python 3.8 vs. Python 3.11, or different versions of CUDA), these packages sit dormant on the system until you explicitly ask for them.

Key Commands to Know:

• module avail: Displays a massive list of all software packages pre-compiled on the cluster. You can filter this by adding a keyword, such as module avail python or module avail gcc.
• module load <package>: Injects the chosen software into your current terminal session. For example, running module load python/3.10.8 updates your environment so that typing python points directly to that specific version.
• module list: Shows you which modules are currently active in your session.
• module purge: Clears out all loaded modules, giving you a clean slate. This is incredibly useful if two software packages are conflicting with one another.

Why you should use them:

Modules are pre-compiled specifically for the cluster’s underlying hardware. This means they are often optimized to run significantly faster than a generic version you download off the internet. Best of all, they take up exactly zero bytes of your personal storage quota. Always check module avail first.

Level 2: Conda / Miniconda (The User-Space Way)

What happens when you need a Python library or a niche scientific package that isn’t in the module list? For most data scientists and researchers, the immediate answer is Conda.

Conda allows you to create completely isolated “sandboxes” (environments) where you can install any package version you want entirely within your user account. However, Conda comes with two massive pitfalls that frequently break beginners’ workflows.

Pitfall #1: The Home Directory Trap

By default, when you run conda create or conda install, Conda downloads and extracts files into a hidden folder in your home directory (~/.conda or ~/.cache).

On almost all HPC clusters, your home directory (/home/username) has a strict, tiny storage quota (often between 10GB and 50GB) intended only for configuration files and source code. Because modern data science environments (especially those involving PyTorch or TensorFlow) can easily balloon to 10GB+ per environment, running a few installations can completely max out your storage quota. When this happens, you won’t even be able to log in or run basic commands.

The Fix: Redirect Conda to Storage or Scratch

Before installing anything with Conda, tell it to store its environments and package caches in your lab’s high-capacity group folder or a fast scratch directory. You can do this by creating or editing a configuration file named .condarc in your home directory:

# ~/.condarc
envs_dirs:
  - /scratch/username/conda/envs
  - /project/labname/shared_conda/envs
pkgs_dirs:
  - /scratch/username/conda/pkgs

Replace /scratch/username/ or /project/labname/ with the actual high-capacity data paths provided by your cluster administrators.

Pitfall #2: Compiling on the Login Node

When you log into a cluster, you land on a “login node.” This node is a shared hallway where hundreds of users are editing files and submitting jobs.

Running a heavy command like conda env create -f environment.yml fires up intense CPU and memory processes to resolve dependencies and compile code. Doing this on a login node will slow down the terminal experience for everyone else on the cluster and will usually trigger an automated system script that kills your process mid-installation.

The Fix: Use an Interactive Compute Session

Whenever you need to build or modify a Conda environment, request a temporary compute node using SLURM’s interactive mode:

# Request an interactive shell on a compute node for 1 hour
srun --tasks=1 --cpus-per-task=2 --mem=8G --time=01:00:00 --pty bash

Once your terminal transfers you to a compute node (e.g., user@node042), you can safely run your Conda installation commands without affecting anyone else.

Level 3: Containers (The Bulletproof Way)

Sometimes, even Conda isn’t enough. If your workflow requires a complex web of system libraries, an entirely different operating system (like a specific version of Ubuntu when the cluster runs Rocky Linux), or you want to replicate an exact software pipeline from a published paper, you need containers.

In the commercial tech world, Docker is the king of containers. However, Docker is fundamentally incompatible with shared HPC clusters because running Docker requires a background process (daemon) with root privileges. If you could run Docker on a cluster, you could easily bypass security and access other users’ data.

Apptainer / Singularity to the Rescue

To solve this, the HPC community developed Apptainer (formerly known as Singularity). Apptainer allows you to run containers completely in “user space” without needing sudo.

Instead of dealing with background daemons, Apptainer compresses an entire container environment into a single, portable file ending in .sif (Singularity Image Format).

Key Commands to Know:

1. Download and Convert a Docker Image: You can pull almost any public image directly from Docker Hub, and Apptainer will automatically translate it into an HPC-safe .sif file:

   apptainer pull ubuntu_python.sif docker://python:3.11-slim
   

2. Execute a Command Inside the Container: To run your code using the software isolated inside that container image, use exec:

   apptainer exec ubuntu_python.sif python3 my_script.py
   

3. Run an Interactive Shell Inside the Container: If you want to look around inside your containerized operating system environment:

   apptainer shell ubuntu_python.sif
   

Why you should use them:

Containers are the ultimate tool for scientific reproducibility. Once you find a container configuration that works, that .sif file will run exactly the same way on your university cluster, a national lab supercomputer, or a cloud instance five years from now.

Summary: The Software Management Checklist

To keep your account active and the cluster running smoothly, run through this mental checklist every time you need new software:

1. Check module avail first. If it’s already installed by the system administrators, load it and move on.
2. Check your storage quotas. Run your cluster’s quota check command (like myquota or lfs quota) to ensure your home directory isn’t on the verge of filling up.
3. Configure your .condarc file. Never leave Conda on its default settings; always point your environment and package directories to a high-capacity scratch or group folder.
4. Never install software on a login node. Use srun to spin up an interactive compute session before compiling or running extensive installers.
5. Use Apptainer for complex pipelines. If a workflow requires intricate system dependencies or absolute reproducibility, skip the installation headaches entirely and look for a container image.

When you log into a university or national lab cluster, you aren’t logging directly into the powerful machines that run your code. Instead, you land on a “login node” shared with dozens of other users. To actually run your heavy computations, you have to request permission from a gatekeeper.

On most High-Performance Computing (HPC) systems today, that gatekeeper is SLURM (Simple Linux Utility for Resource Management). Think of SLURM as a reservation system for a supercomputer. While it has a reputation for being complex, most day-to-day work comes down to a handful of basic commands. This guide covers the essential subset you need to get started: submitting jobs, monitoring them, stopping them when things go sideways, and understanding why they might be stuck in line.

One major caveat: SLURM is highly configurable, and every institution customizes it differently. Queue names, time limits, memory defaults, and specific resource rules vary from cluster to cluster. The commands here are standard SLURM, but you should always cross-reference them with your local site’s documentation.

Submitting a Job

You don’t run heavy scripts directly in the terminal on an HPC. Instead, you wrap your commands in a shell script and hand it over to SLURM using the sbatch command:

sbatch my_job.sh

When SLURM accepts your job, it assigns it a unique Job ID and prints it to the screen:

Submitted batch job 847312

You will use this number to track, manage, or cancel your job. A minimal, beginner-friendly job script looks like this:

#!/bin/bash
#SBATCH --job-name=hpc-demo
#SBATCH --output=hpc-demo-%j.out
#SBATCH --error=hpc-demo-%j.err
#SBATCH --time=04:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G

# The commands below will run on the compute node once allocated:
echo "Starting my analysis..."
python3 my_analysis_script.py --input data.csv

The special #SBATCH lines at the top tell SLURM exactly what resources your code needs to run.

What do these headers mean?

• --job-name: A friendly nickname for your job so you can spot it easily in the queue.
• --output and --error: Where your program’s text output and error messages should be saved. The %j is a variable that automatically fills in your unique Job ID, preventing different runs from overwriting each other’s logs.
• --time: The maximum runtime allowed (Hours:Minutes:Seconds). If your job exceeds this, SLURM will stop it automatically.
• --cpus-per-task and --mem: The requested number of CPU cores and total RAM (8 Gigabytes in this case).

Passing Arguments: Header vs. Command Line

Every #SBATCH line inside your script is identical to passing a flag directly to sbatch on the command line. These two methods accomplish the exact same thing:

# Option A: All configuration is kept inside the script header
sbatch my_job.sh

# Option B: Overriding the script's defaults from the command line
sbatch --time=08:00:00 --mem=16G my_job.sh

Command-line flags always take precedence over header directives. The best habit to build is putting your sensible defaults in the script header—the values you expect to use 90% of the time—and then overriding them on the command line for one-off experiments that require extra memory or longer runtimes.

Watching Your Jobs

Once you submit a job, squeue is the command you will use most often. Running it without arguments shows every single job currently running on the entire cluster, which is overwhelming. Instead, filter it to show only your jobs:

squeue -u $USER

The output will look something like this:

JOBID    PARTITION  NAME        USER     ST   TIME   NODES  NODELIST(REASON)
847312   general    hpc-demo    dthain   R    0:42   1      node047
847313   general    hpc-demo    dthain   PD   0:00   1      (Resources)
847314   general    hpc-demo    dthain   PD   0:00   1      (Priority)

The ST (State) column tells you what your job is currently doing:

• R (Running): Your job is actively executing on a compute node.
• PD (Pending): Your job is waiting in line.

The NODELIST(REASON) column explains why a pending job hasn’t started yet. (Resources) means the cluster is currently full and waiting for other users’ jobs to finish. (Priority) means your job is sitting behind higher-priority work in the queue.

If you want to check when SLURM expects your job to start, add the --start flag:

squeue -u $USER --start

Note: SLURM’s start-time estimates are notoriously optimistic, as they assume every running job will use its maximum requested time. Treat these numbers as rough ballparks rather than guarantees.

Cancelling a Job

If you realize your script has a bug, or you accidentally requested the wrong parameters, you can clear it out of the system using scancel and its Job ID:

scancel 847312

To wipe the slate clean and cancel all of your jobs at once:

scancel -u $USER

If you only want to clear out your waiting jobs while letting your active, running jobs finish safely, specify the pending state:

scancel -u $USER -t pending

Checking Cluster Availability

Before you submit a massive job, it is helpful to see how busy the cluster is. The sinfo command provides a snapshot of the available hardware pools (called “partitions”):

sinfo

PARTITION  AVAIL  TIMELIMIT   NODES  STATE   NODELIST
general*   up     7-00:00:00  12     idle    node[001-012]
gpu        up     2-00:00:00  4      mix     gpu[01-04]
debug      up     0-01:00:00  2      idle    node[013-014]

Pay attention to the STATE column:

• idle: The nodes are completely free and ready for work.
• mix: Some CPU cores on these nodes are busy, but others are still available.
• alloc: The nodes are completely full.
• drain: The nodes have been taken offline by administrators for maintenance.

Inspecting a Live or Stuck Job

For detailed troubleshooting on a specific job, scontrol show job will output a comprehensive diagnostic record:

scontrol show job 847312

The output is quite long, but look closely for JobState, Reason, and TRES (the exact resource request). If a job has been pending for days, the Reason field will often reveal if it is blocked by an administrative hold, an impossible resource request, or standard queue traffic.

Checking What Completed Jobs Actually Used

One of the biggest mistakes beginners make is overestimating how much memory or time their code needs. Requesting too many resources forces your job to wait in line much longer than necessary.

To look back at a completed job and see its actual resource footprint, use sacct:

sacct -j 847312 --format=JobID,JobName,State,Elapsed,MaxRSS,ReqMem

The MaxRSS field shows the peak amount of RAM your job actually consumed during its run. Compare this against ReqMem (what you requested). If your job only used 2GB of RAM but you requested 32GB, you are artificially delaying your own queue times. Aim to request roughly 20–30% more memory than your historical peak to keep your queue times short while avoiding out-of-memory errors.

A Few Silent Pitfalls to Avoid

SLURM enforces local cluster rules strictly, and it often does so silently. Keep these four items in mind as you review your institution’s specific user guide:

• Partition Time Caps: Most cluster queues have hard maximum runtimes (e.g., a 24-hour limit). If you ask for 48 hours on a 24-hour queue, SLURM will reject your job the second you attempt to submit it.
• Account Flags: Many institutions require you to specify a billing project or account via --account=my_lab_group. If you omit this, your job may fail instantly or be assigned to a lowest-priority pool.
• Silent Memory Defaults: If you don’t specify a --mem flag, SLURM will assign you a default value per core. On some clusters, this default can be incredibly small (like 1GB), causing your program to crash unexpectedly with an OUT_OF_MEMORY error. Always declare your memory explicitly.
• Job Arrays for Repetitive Work: If you need to run the exact same analysis script over 50 different data files, do not run sbatch 50 separate times. Instead, look into Job Arrays using sbatch --array=1-50. It allows the scheduler to handle your workload as a single cohesive unit, saving your sanity and keeping the cluster running smoothly.

Debugging distributed systems on HPC has its own rhythm. The feedback loop is slow, the machines are shared, and the error you are chasing may only appear under load, across nodes, or after a scheduler delay you cannot reproduce locally. This note grew out of issue #4415: a segmentation fault (SIGSEGV) in task_min_resources inside work_queue.c, triggered by Makeflow running a large parallel workflow in Work Queue mode.

The habits below are less about any single tool and more about discipline: keeping your changes visible, your experiments small, and your teammates in the loop. They apply equally whether you are working in work_queue.c or anywhere else in a large C codebase running on a shared cluster.

Talk to your teammates early and often

The most underrated debugging tool is a colleague. When you hit a wall the instinct is to keep grinding alone until you have something to show. Resist it. A two-sentence description of what you changed and where you are stuck surfaces assumptions you did not know you were making before you spend a day looking in the wrong function.

Post a short note to your team channel whenever you switch gears. If you are about to run a batch of SLURM jobs to test a hypothesis, say so. Someone may have already run that experiment, or may know that the catalog tick interval is configurable and could be shortened for testing.

The flip side: when someone asks for help, give them the full picture up front. Paste the Valgrind output, the backtrace, and a short description of what you have already tried. “Here is what I know” cuts the back-and-forth in half.

Put your changes somewhere others can see them

Debugging on HPC usually means modifying source, rebuilding, and staging a binary on a shared login node where the next person has no idea what you changed. The fix is obvious but easy to skip when you are in a hurry: commit and push early, to a branch, a fork, or a scratch repository your colleagues can actually reach.

git checkout -b debug/task-min-resources-segfault
# edit work_queue/src/work_queue.c
git add work_queue/src/work_queue.c
git commit -m "add null check before task_min_resources dereferences ready list node"
git push origin debug/task-min-resources-segfault

A pushed branch does several things at once. It gives collaborators a URL to look at instead of a terminal paste. It lets them check out your exact tree, reproduce the build, and point out something you missed. And it is an automatic checkpoint: if make clean goes sideways or a node corrupts your scratch directory, the work is not gone. Uncommitted changes on a cluster login node are one bad disk event away from disappearing.

Keep a running log of what you changed and why

Segfault hunts often take days or weeks. Memory is not reliable across that span. Keep a plain text file, a section in the PR description, or a lab notebook — and update it every time you try something:

2026-06-09  opened #4415; Valgrind log attached; fault site is work_queue.c:8174 in task_min_resources
2026-06-10  added null check on q->ready_list head before dereferencing — crash still appears on run 3 of 5
2026-06-11  hypothesis: node is removed from the ready list but list_next still returns it; added fprintf around list traversal
2026-06-12  confirmed: stale pointer survives across catalog tick; looking at where nodes are freed vs unlinked

This log is not for posterity. It is for you, tomorrow morning, when you cannot remember why you commented out that block. It is also what you paste when you ask a teammate for help.

Scale down before you scale out

Submitting a five-hundred-task Makeflow workflow to test a one-line fix is one of the most reliable ways to waste an afternoon. Queue wait times are unpredictable, the logs from hundreds of workers are noisy, and if the fix is wrong you have burned allocation budget and still do not know why.

The better approach is to find or construct the smallest possible reproducer. Once the small case reproduces reliably, you have a debuggable target. Only when that case is clean do you run at scale to confirm nothing regresses. This discipline also makes it far easier to share a reproducer with a teammate: “clone, run these two commands, watch it crash” is a much better bug report than “submit two hundred SLURM jobs and search the logs.”

Instrument the code before you reach for a debugger

Before attaching GDB or running Valgrind, the fastest thing you can do is add targeted fprintf calls to verify that your fix is actually being reached. This sounds obvious, and yet the most common source of “my patch does nothing” confusion is that the code path you edited is never entered for the input you are testing.

/* In work_queue.c, before the dereference that Valgrind flagged */
fprintf(stderr, "[DEBUG] task_min_resources: checking node %p on ready list\n", (void *)t);
if (!t) {
    fprintf(stderr, "[DEBUG] task_min_resources: null node encountered — skipping\n");
    continue;
}
fprintf(stderr, "[DEBUG] task_min_resources: node task_id=%d resources_requested=%p\n",
        t->taskid, (void *)t->resources_requested);

Write to stderr, not stdout, which may be buffered or redirected by the framework. Add prints before and after the critical section. If the “before” line appears but the “after” line does not, you found the crash site. If neither appears, the function is not being called — which means the bug is upstream of where you thought it was, and you just saved yourself an hour of reading the wrong code.

If the prints confirm your fix is being reached but the crash keeps happening, the root cause is elsewhere: probably the point where the stale node enters the list, not the point where it is dereferenced. Move the instrumentation upstream.

Gate these prints behind a compile-time flag before merging so they do not pollute production builds:

#ifdef WQ_DEBUG_TASK_RESOURCES
fprintf(stderr, "[DEBUG] task_min_resources: node %p task_id=%d\n",
        (void *)t, t->taskid);
#endif

Pass -DWQ_DEBUG_TASK_RESOURCES in CFLAGS to turn them on; a normal build leaves no noise.

Reach for GDB and Valgrind when instrumentation is not enough

When fprintf tells you where the crash is but not why, it is time to bring in a real debugger. For a crash you can reproduce locally, GDB is the first stop:

# Build with debug symbols and no optimization so variables are readable
./configure CFLAGS="-g -O0"
make -C work_queue/src
 
# Run until it crashes, then inspect the frame
gdb --args work_queue_worker -d all localhost 9123
(gdb) run
(gdb) bt                        # full backtrace at the crash point
(gdb) frame 2                   # switch to the frame inside task_min_resources
(gdb) p t                       # is the pointer null or garbage?
(gdb) p t->resources_requested  # does the struct look sane?
(gdb) watch t->taskid           # set a watchpoint to catch when this field changes

Valgrind slows execution by roughly ten to twenty times, so pair it with the scaled-down reproducer from the previous section. A handful of short tasks that trigger the ready-list traversal during a catalog tick is a much better target than a full production workflow.

One practical note: you generally cannot run GDB interactively inside a SLURM job. If the bug only appears at scale, use Work Queue’s -d all flag to write a verbose trace to a shared filesystem path that both the execute node and your login node can read:

work_queue_worker -d all -o /cephfs/group/myproject/wq-logs/worker-$(date +%Y%m%d%H%M%S).log \
  localhost 9123

Then reconstruct a local reproducer from what the trace tells you, and use GDB on that.

Read the logs that are already there

Before adding instrumentation or firing up a debugger, check what Work Queue already records. The -d all flag produces a structured trace across subsystems — scheduling decisions, catalog updates, task state transitions — that often answers the question before you write a single fprintf.

# Watch the debug stream in real time on a local test run
work_queue_worker -d all localhost 9123 2>&1 | grep -i "task_min\|ready\|segfault\|null\|error"
 
# After a cluster run, search a saved log
grep -n "task_min_resources\|ready_list" worker-20260609.log | head -40

The log format uses timestamps and subsystem tags, so you can narrow a twenty-thousand-line trace to the thirty lines around the catalog tick event without much effort. For the #4415 bug in particular, looking for the catalog update message immediately before the crash narrows the search window considerably.

Turn on every debug flag and log everything

When you are stuck, err heavily on the side of logging too much rather than too little. A verbose log that you have to grep through is far better than a quiet one that leaves you guessing. The more you record, the more likely it is that the crash site, the bad pointer, and the sequence of events leading to it are all sitting in the file waiting for you.

CCTools exposes this through the -d flag, which accepts one or more subsystem names. The relevant ones for a Work Queue + Makeflow investigation are wq (Work Queue task scheduling and worker communication), batch (the batch system layer that submits SLURM or Condor jobs), rmon (the resource monitor that measures cores, memory, and disk per task), and makeflow (the Makeflow DAG engine, which covers parsing, lexing, and the run loop). Pass them all:

# Worker: turn on every relevant subsystem and save to a named file
work_queue_worker -d all -o worker.log localhost 9123 &
 
# Makeflow manager: same idea, log to a separate file
makeflow -T wq -d wq,batch,rmon,makeflow -o makeflow.log mini.makeflow

-d all is the blunt instrument — it sets every bit in the flag word and includes subsystems you probably do not care about on this run. That is fine. The extra volume costs you nothing except disk space, and the subsystem tags in the log make it easy to filter later. If the log grows large enough to be unwieldy, narrow it down once you know which subsystem to focus on; start with all and restrict from there.

On the cluster, where you cannot read stderr interactively, always pair -d all with -o pointing at a shared filesystem path that your login node can reach after the job finishes. Give each job a unique filename so concurrent workers do not overwrite each other:

work_queue_worker -d all \
  -o /cephfs/group/myproject/wq-logs/worker-$(date +%Y%m%d%H%M%S)-$$.log \
  localhost 9123

The $$ expands to the worker process id on the execute node, which is a cheap uniquifier on top of the timestamp. Once the logs land, search them as a unit:

# Find the catalog tick and the crash window across all workers in a session
grep -h "task_min_resources\|catalog\|ready_list\|SIGSEGV" \
  /cephfs/group/myproject/wq-logs/worker-20260609-*.log \
  | sort -k1,2 | less

The -h flag suppresses the filename prefix so the timestamps sort cleanly. Sorting by the first two fields — date and time — stitches the per-worker logs into a single chronological view of what the whole pool was doing in the seconds before the crash. That cross-worker timeline is often what finally makes the bug obvious.

One warning: do not turn on --debug-workers in work_queue_factory and also pass -d all through --extra-options at the same time. The factory-side flag already appends -d all -o worker.<n>.log to every submission; doubling up means every worker writes two interleaved logs to slightly different paths and you end up reading redundant output. Pick one approach per session.

The quick checklist before every cluster run

Before you fire off a batch job, spend sixty seconds on these:

• Did you rebuild and reinstall after your last edit? A stale binary is the most common source of “my fix does nothing.”
• Is the debug log going to a path that exists and is writable from the execute node?
• Is the job small enough that you will get a result in under ten minutes, or do you have a good reason to go bigger right now?
• Did you commit your current state so that whatever happens on the cluster, the code is recoverable?
• Did you tell a teammate what you are about to test, so they can flag it if they know something relevant?

None of these take long. Together they prevent the most common ways a cluster debugging session turns into a wasted afternoon.

Last week, I had the opportunity to attend Throughput Computing Week 2026 in Madison, Wisconsin, organized by the Center for High Throughput Computing (CHTC). I participated in the conference on Wednesday and Thursday and came away with a deeper appreciation for the diverse ways high-throughput computing is impacting research across disciplines.

One of the highlights of the event was attending my advisor Dr. Douglas Thain’s talk on Thursday, “Wrangling Massive Task Graphs with VineReduce.” The presentation showcased new approaches for managing large-scale computational workflows and demonstrated the continued evolution of distributed computing systems.

Beyond the technical sessions, one of the most rewarding aspects of the conference was meeting researchers from a wide range of scientific domains. I spoke with teams working in space research who use HTCondor to process massive amounts of satellite-generated terrain data. Other researchers shared how high-throughput computing is accelerating genome analysis in biology and enabling large-scale weather modeling that helps civil engineers design more efficient roads and buildings.

Among the many talks, I was particularly fascinated by discussions around Pelican and its role in distributed data storage and access. I also learned about emerging challenges surrounding next-generation hardware accelerators and AI-focused chips. Researchers and computing teams from universities across the country discussed how they are balancing growing computational demands through cloud resources, strategic infrastructure investments, and cost optimization techniques.

As artificial intelligence continues to reshape science and engineering, the conference highlighted how computing infrastructures are evolving alongside it. From data-intensive scientific discovery to large-scale AI workloads, the conversations at Throughput Computing Week reinforced how rapidly the world of research computing is changing—and how important collaboration across disciplines will be in navigating that future.

This event brought together experts from 10 different experiments to align on shared challenges and collaborate on community solutions. Beyond establishing a common technical language, we mapped out key focus areas for the future:

• Resource optimization: Improving execution behavior and utilization.
• Data-aware late binding: Navigating data locality and streaming.
• Request, provenance & validation: Exploring unified standards for workflow interfaces.
• AI in operations: Leveraging AI assistance in complex distributed systems.

The conversation is just getting started. If you’re interested in the evolution of Workflow/Workload Management and community standards, we’d love for you to join us! Connect via our Mattermost channel

This May CCL third-year PhD student Jin Zhou traveled to New Orleans for IPDPS 2026, the 40th IEEE International Parallel & Distributed Processing Symposium, held at the Marriott on Canal Street from May 25 to 29. He presented our paper SciWIND: Effectively Exploiting Node-Local Storage for Data-Intensive High-Energy Physics Workflows, which looks at how to use node-local scratch more deliberately when large HEP workflows run on opportunistic clusters and workers fail mid-run. The talk was a nice cap on a line of work the lab has been pushing through TaskVine and our HEP collaborations, and it was good to put the system in front of people who live with scheduling, storage, and workflow engines every day.

Between sessions Jin followed the conference program: tutorials and workshops on the first two days, then the main track, keynotes, and plenty of hallway conversations. IPDPS still has that familiar mix of parallel algorithms, distributed systems, and applications at scale. AI was clearly a hot topic this year, showing up in keynotes, panels, and hallway chats about training and inference at scale. A few questions after the SciWIND talk turned into longer conversations about eviction recovery, disk pressure on shared filesystems, and where workflow runtimes should own policy versus leave it to the user. New Orleans helped too, with late walks along Canal Street and coffee between sessions that made the week feel less like a sprint and more like a real meeting of the community.

Thanks to everyone who came to the SciWIND session and to the IPDPS organizers for another smooth run. We are glad the paper is out in the proceedings and happy to keep the conversation going with groups wrestling with the same storage and resilience headaches in production science workflows.

This week our lab was in Chicago for GCASR 2026—the Greater Chicago Area Systems Research Workshop (gcasr.org), a regional gathering focused on computer systems, from operating systems and distributed infrastructure to the tools that make large-scale science tractable. It is a friendly venue for students and faculty to share work in progress, compare notes on real systems problems, and meet peers from departments and labs around the Midwest.

The CCL makes this trip every year: Chicago is home base for GCASR, and showing up has become part of how we stay plugged into the systems community between the bigger conference cycles.

Each student on the team presented a research poster, as we typically do at this venue. The poster floor was busy, with good questions, fast feedback, and plenty of hallway conversations that do not fit on a slide deck. Between sessions, people attended invited talks and panels, swapped implementation details, and followed threads from scheduling and storage to workflows and AI-facing infrastructure.

Thanks to everyone who stopped by our posters and to the organizers for another well-run GCASR. We appreciated the insights, the introductions, and the chance to catch up with colleagues in person. We look forward to seeing familiar faces at GCASR 2027!