In a new paper presented at IEEE eScience 2026, Md Saiful Islam along with co-authors Reyer Band, Jin Zhou, Tanu Malik, Kevin Lannon, and Douglas Thain address one of the most stubborn obstacles in computational science: data portability across High-Performance Computing (HPC) systems.

In their paper, Declare, Resolve, Reuse: A Portable Data Layer for Notebook Workflows Across HPC Systems, Saiful introduces an externalized, declarative data layer built for the Floability workflow engine. This framework expands on the “backpack” packaging abstraction, allowing data-intensive Jupyter notebooks to run seamlessly across disparate HPC environments without code modifications.

The Problem: The Data Portability Gap

Jupyter notebooks are ubiquitous in exploratory science, yet studies reveal that up to three-quarters of published notebooks fail when re-executed end-to-end. While software environments are increasingly containerized or packaged with tools like Conda and Apptainer, data handling remains fragile.

As highlighted in Figure 2 (above), scientific notebooks routinely embed site-specific assumptions into Python cells:

• Hard-coded absolute file paths (e.g., /scratch/data...) that break when transferred to another cluster.
• Ad-hoc download cells that re-fetch gigabytes of data on every single execution.
• Unset or site-dependent environment variables.

When a researcher attempts to move a notebook from Site A to Site B or Site C, the notebook code must be manually edited and rewritten into distinct, fragile variants.

The Solution: Declare, Resolve, Reuse

Saiful’s approach decouples data requirements from notebook code by introducing an explicit specification file (data.yml) managed by the Floability runtime. The architecture relies on three core tenets:

1. Declare: Workflows define input datasets, logical names, expected validation criteria (such as file size and SHA-256 checksums), and target runtime paths in a declarative YAML file. Profiles allow users to switch between dataset variants (e.g., train_set vs. test_set) while serving data to the notebook at the exact same relative target path.
2. Resolve: Floability abstracts access across heterogeneous sources—including Pelican federated storage, S3 object stores, HTTP endpoints, and local filesystems—with support for multi-source fallback policies.
3. Reuse: An HPC-local cache indexes datasets using a combination of the specification schema key and remote source metadata (such as ETag, modification time, and file size). Unchanged datasets are identified instantly without downloading full contents, letting repeat runs skip data transfers safely.

Multi-Site Evaluation

To evaluate the portable data layer, Saiful tested three large-scale, data-intensive workflows:

• DV5: A High-Energy Physics (HEP) analysis running over ~108 GB of simulated LHC data across 44,010 tasks.
• LFV: A Lepton Flavor Violation analysis over ~7.5 GB of data.
• DConv: An image convolution workflow processing ~93 GB of GOES-16 satellite imagery across 48,528 tasks.

The evaluation deployed identical Floability backpacks across three distinct HPC facilities: ND CRC (HTCondor scheduler), Purdue Anvil (Slurm scheduler), and TACC Stampede3 (Slurm scheduler).

Key Findings:

• Zero Code Edits: The exact same notebook code ran completely unchanged across all three systems, with site-specific configurations supplied strictly via CLI options.
• Minimal Lookup Overhead: Metadata-based cache key lookups took ~0.12 seconds for single files (up to 1 GB) and remained under 4.3 seconds even for complex dataset directories containing 10,000 files.
• Streamlined Execution: On repeat executions, cached dataset reuse completely removed data materialization and software environment creation from the critical runtime path.

By taking data movement and staging logic out of individual notebook cells, Saiful’s work turns portable, repeatable scientific execution from an ad-hoc engineering chore into an automated property of the workflow.

Resources

• Paper PDF: Declare, Resolve, Reuse: A Portable Data Layer for Notebook Workflows Across HPC Systems
• Project Website: floability.github.io
• Artifacts & Benchmarks: GitHub Repository

This week, Dr. Thain, Alan Rodrigues, and Barry are attending the eScience’26 conference in Naples, Italy. The conference is being held at the Villa Doria d’Angri, an architectural masterpiece belonging to the University of Naples “Parthenope,” from September 28 - October 2, 2026.

Barry is presenting his work on: Fusion and Reshaping of Hidden Task Graphs for High Energy Physics Applications.

Alan is presenting his work on: Hybrid Workflow Composition for Extreme-Scale Data Processing: A Case Study on the HL-LHC.

Dr. Thain is presenting (on behalf of Saiful): Declare, Resolve, Reuse: A Portable Data Layer for Notebook Workflows Across HPC Systems.

Last Tuesday, September 15, our Safety-Aware Drone Ecosystem (SADE) team participated in a NASA University Leadership Initiative (ULI) presentation titled “Who Should Enter? Reputation, Risk, and Decision Making for UAVs in Low-Altitude Airspace.” The broader SADE project, led by Prof. Jane Cleland-Huang, brings together researchers from the University of Notre Dame and other collaborating institutions to explore how small uncrewed aircraft systems can safely operate in managed airspace through fair, transparent, explainable, and consistent decision-making. The presentation highlighted several components of the ecosystem, including safety certification, proving-ground evaluation, runtime monitoring and reputation models, system architecture, and simulation.

During the presentation, Prof. Douglas Thain introduced the key components of the platform and its goal of providing a self-service environment for evaluating custom software pilots interacting with SADE Zones under challenging and realistic conditions. The platform integrates PX4, Gazebo, Unreal Engine, and realistic geographic environments, enabling researchers to create complex scenarios involving multiple drones, terrain, buildings, wind conditions, and dynamically managed airspace. The presentation highlighted several case studies developed by our team, including the Utah Mountains simulation with 40 drones, Yosemite Search and Rescue, Indianapolis Airport, and Chicago Downtown scenarios.

We also showcased videos and screenshots demonstrating different capabilities of the platform. These included drones requesting permission to enter a SADE Zone, rerouting around the zone when access was denied, experiencing the effects of realistic wind conditions generated for the simulation, and interacting with real terrain and buildings—including examples where a drone collides with a building in Chicago. Together, these demonstrations showed how SADE Simulation can be used to study drone behavior and software-pilot decisions in realistic, safety-critical environments.

It was rewarding to see the simulation work presented alongside the other components of SADE, demonstrating how the different research efforts come together toward a common goal: enabling safer and more trustworthy drone operations in increasingly complex low-altitude airspace. The talk was also a great reminder of how our day-to-day engineering work contributes to the broader NASA ULI research effort and the future of safe autonomous aviation.

This week, CCL fourth-year PhD student Md Saiful Islam and second-year PhD student Ryan Hartung traveled to Chicago for APeX Summit 2026, Academy, Parsl, Globus Compute, held at the University of Chicago’s Data Science Institute from September 14–15, 2026. Saiful presented his work Floability Backpacks: Packaging Workflows for Portable HPC Deployment.

This week, Ben, presented his work on the Vine CMS analysis stack during the US CMS Analysis Facilities monthly meeting.

The Vine CMS analysis stack is a reference stack for running CMS analysis workflows with Coffea on top of TaskVine, orchestrated by VineReduce. It does not depend on Notre Dame resources, so it can work anywhere there is a Python environment and either local cores or a batch cluster (HTCondor, SLURM, SGE, …) to point workers at.

You can view his presentation slides HERE.

Shared network filesystems, CephFS, Lustre, or NFS, are the backbone of high-performance computing clusters. They allow hundreds of compute nodes to seamlessly read and write to the same central directories simultaneously.

However, shared parallel storage operates under vastly different physical constraints than a local laptop SSD. When users treat shared network storage like a personal hard drive, they often encounter mysterious slowdowns, hit unexpected quota limits, or bring storage responsiveness to a crawl for the entire cluster.

The Small File Problem: Why 100,000 Small Files Destroy Performance

Parallel network filesystems are optimized for high bandwidth throughput, meaning they excel at streaming massive, multi-gigabyte files (like large dataset archives or continuous checkpoint files) across dedicated network hardware.

They perform poorly, however, when handling small file I/O operations.

Metadata Bottlenecks Explained

Every file creation, directory listing, read, write, and deletion request requires a communication step with the filesystem’s Metadata Server (MDS). The MDS tracks file ownership, directory trees, permissions, and physical storage locations (inodes).

• Writing one 10GB binary file requires 1 metadata request to allocate space, followed by continuous high-speed data streaming.
• Writing 100,000 tiny 100KB log files requires 100,000 individual metadata requests, forcing the MDS server to handle constant network handshakes and lock updates.

When a batch job generates thousands of tiny intermediate files or reads an uncompressed dataset of tiny images directly from shared storage, the metadata server becomes overloaded. This creates massive I/O latency, causing job execution to stall and slowing down directory navigation cluster-wide.

Decoding Storage Quotas: Capacity vs. Inodes

When a job fails with Disk quota exceeded or No space left on device, your immediate instinct is to check your disk space usage. But cluster storage quotas track two completely separate limits:

1. Block Quota (Capacity): The physical amount of raw data storage allocated to your user or project (measured in GB or TB).
2. Inode Quota (File Count): The total number of individual files and directories allowed in your space, regardless of file size.

Creating hundreds of thousands of small logs, virtual environments, or temporary output files will max out your Inode quota long before you consume your block storage allocation. Once your inode limit is reached, the operating system blocks the creation of new files—even if you have terabytes of unused disk space remaining.

Leveraging Node-Local /tmp Space Safely

To bypass network latency, protect metadata servers, and prevent quota exhaustion, heavy file operations should take place on node-local storage (such as fast local NVMe or SSD drives mounted at /tmp or $TMPDIR).

Node-local storage sits directly inside the compute node hosting your job, bypassing the shared network network entirely.

The Recommended Node-Local Workflow Pattern:

Structure your submission script to stage files locally during job execution:

#!/bin/bash
#SBATCH --job-name=fast_io_job

# 1. Define a temporary directory unique to this job on local node storage
LOCAL_DIR="/tmp/$USER/$SLURM_JOB_ID"
mkdir -p "$LOCAL_DIR"

# 2. Copy and uncompress input datasets from central storage to local storage
cp /scratch/username/dataset.tar.gz "$LOCAL_DIR/"
tar -xzf "$LOCAL_DIR/dataset.tar.gz" -C "$LOCAL_DIR/"

# 3. Execute your program using local /tmp paths for input and intermediate output
python3 my_training_script.py --data-dir "$LOCAL_DIR/dataset" --out-dir "$LOCAL_DIR/output"

# 4. Pack final results into a single archive file and move back to shared storage
tar -czf "$LOCAL_DIR/final_results.tar.gz" -C "$LOCAL_DIR/output" .
cp "$LOCAL_DIR/final_results.tar.gz" /scratch/username/results/job_${SLURM_JOB_ID}_results.tar.gz

# 5. Clean up local node space before exiting
rm -rf "$LOCAL_DIR"

Summary: The Cluster Storage Hygiene Checklist

1. Avoid uncompressed datasets. Bundle datasets into a single archive (.tar, .zip, .h5, or .sqlite) before transferring them to shared storage.
2. Monitor both quota metrics. Check both your byte allocation and file count allocations using your cluster’s quota tools (e.g., myquota or ceph df).
3. Stage temporary files on /tmp. Use local node storage ($TMPDIR or /tmp) for fast read/write operations during execution.
4. Aggregate output logs. Buffer runtime output in memory or write progress to a single structured log file rather than generating individual files per iteration.
5. Always clean up local directories. Include explicit cleanup commands (rm -rf) in your job script to ensure local scratch disk space stays clean for subsequent users.

As campus comes back to life for the start of the new semester, the Cooperative Computing Lab is excited to hit the ground running with fresh research, ongoing projects, and growing software initiatives.

Welcoming Andrés Iglesias to the Lab

We are thrilled to re-welcome Andrés Iglesias back to the lab! Andrés will be resuming his work on the Pledge project in close collaboration with Colin. We are excited to see the impact of their continued work.

Team Roles & Responsibilities

To keep our workflows running smoothly across research, releases, and community engagement, our internal team coordinator roles remain steady for the semester:

• Lax: Team Meeting Organizer (keeping our weekly meetings structured and productive)
• Ryan: Communications Coordinator (creating blog posts, managing lab updates, and announcements)
• Saiful: Outreach & Training Lead (organizing tutorials and helping new users navigate our tools)
• Ben: Release Engineer (ensuring stable, high-quality software builds and deployments)

Here’s to a productive semester of research, collaboration, and open-source computing! Stay tuned for more technical articles and project updates in the coming weeks.

Working over SSH on a remote supercomputer means your active connection is always at the mercy of your network. A brief Wi-Fi drop, a VPN hiccup, or closing your laptop lid will abruptly close your SSH session—killing whatever interactive build, environment setup, or monitoring script you had running in that terminal window.

Beyond connection drops, navigating massive directory trees and digging through hundreds of job output files with basic commands quickly becomes tedious. Mastering a few quality-of-life Linux tools transforms the remote terminal from an unforgiving interface into a safe, efficient workspace.

1. Terminal Multiplexers: Bulletproof Your Remote Sessions

A terminal multiplexer like tmux (or GNU screen) creates a persistent session on the remote server that runs independently of your SSH connection. If your network disconnects, your remote programs keep running in the background. When you log back in, you simply reattach to the session exactly where you left off.

Essential tmux Commands:

• Start a new named session:

  tmux new -s workflow
  

• Detach from a session (leaves it running in the background): Press Ctrl+b, release, then press d.
• List all active sessions:

  tmux ls
  

• Reattach to a running session:

  tmux attach -t workflow
  

• Kill a finished session:

  tmux kill-session -t workflow
  

Pro Tip: Always launch tmux on a login node before requesting an interactive compute session (srun or qsub). If your Wi-Fi drops mid-job, your interactive compute node allocation stays alive!

2. Searching Fast with find and grep

When running multi-job workflows, manual file inspection breaks down fast. Combining find (to locate files) and grep (to search file text) lets you audit log directories in seconds.

Locate Specific Files with find

Locate files by name, age, or extension across deep directory hierarchies:

# Find all error logs modified in the last 24 hours
find ./logs/ -name "*.err" -mtime -1

# Find all Singularity image files in your scratch space
find /scratch/$USER -name "*.sif"

Search Text Inside Logs with grep

Search through hundreds of output files without opening them individually:

# Search recursively (-r) for "OutOfMemory", displaying line numbers (-n)
grep -rn "OutOfMemory" ./logs/

# Case-insensitive search (-i) across all .err files
grep -i "error" slurm-*.err

Combine Both Tools

Pass matching files directly into grep using the -exec flag:

find ./logs/ -name "job_*.out" -exec grep -H "FAILED" {} +

3. Shell Shortcuts: Custom Aliases in ~/.bashrc

If you find yourself typing the same long cluster paths or complex monitoring commands every day, automate them using shell aliases.

An alias is a custom keyword mapped to a longer shell command. By adding aliases to your shell configuration file (~/.bashrc), they load automatically every time you log in.

Useful HPC Aliases to Add to ~/.bashrc:

# Open ~/.bashrc in your favorite editor (e.g., nano ~/.bashrc) and add:

# Fast directory navigation
alias myscratch='cd /scratch/$USER'
alias myproject='cd /project/labname/$USER'

# Quick scheduler status checks
alias sq='squeue -u $USER'
alias sqstat='squeue --format="%.18i %.9P %.12j %.8u %.2t %.10M %.6D %R"'

# Request a standard 2-hour interactive compute session
alias devnode='srun --cpus-per-task=4 --mem=16G --time=02:00:00 --pty bash'

# Colorized log checking
alias tailerr='tail -n 50 -f *.err'

After editing ~/.bashrc, apply the changes immediately by running:

source ~/.bashrc

Summary: The Shell Survival Checklist

1. Never run critical interactive tasks in a bare SSH shell. Wrap long interactive sessions inside tmux so network drops won’t kill your progress.
2. Use grep -rn to debug log folders instantly. Scan entire directories for error keywords without opening files one by one.
3. Use find to hunt down abandoned files. Clean up old scratch files and large core dumps to keep your storage quota under control.
4. Customize ~/.bashrc with shortcuts. Create aliases for your common cluster paths, job status checks, and interactive node requests.

You submit a batch job, wait in the queue, and suddenly see its status transition to FAILED or HOLD. When you open your output directory, you are greeted by a 200-line wall of obscure error messages, memory dumps, and cryptic exit codes.

It is a classic high-performance computing friction point: feeling overwhelmed by stack traces and assuming something catastrophic went wrong. But HPC error logs are structured breadcrumbs, not punishments. Once you know where your scheduler writes these files and how to parse the output, identifying the root cause takes seconds across any Workload Manager—whether your cluster uses SLURM, Univa Grid Engine (UGE / SGE), or HTCondor.

Standard Output vs. Standard Error vs. Event Logs

When running non-interactive batch jobs, system output streams split into separate files depending on your workload manager:

• Standard Output (.out / stdout): Reserved for standard runtime logs, progress bars, print() statements, and calculation results.
• Standard Error (.err / stderr): Reserved for system warnings, thrown exceptions, segmentation faults, and library errors.
• Workflow Event Log (.log): Unique to schedulers like HTCondor, this file records lifecycle events managed by the scheduler itself (e.g., job submission, node assignment, resource tracking, and hold reasons).

Here is how you define these log files across the three major schedulers:

SLURM (.sub / .sh)

#SBATCH --output=my_job_%j.out
#SBATCH --error=my_job_%j.err

(Where %j inserts the unique SLURM Job ID).

Univa Grid Engine / SGE (.sub / .sh)

#$ -o my_job_$JOB_ID.out
#$ -e my_job_$JOB_ID.err

(Where $JOB_ID inserts the unique Grid Engine Job ID).

HTCondor (.submit)

output = my_job_$(Cluster).out
error  = my_job_$(Cluster).err
log    = my_job_$(Cluster).log

(Where $(Cluster) represents the HTCondor Cluster/Job ID).

When a job fails, always inspect the .err file first (and check the .log file if you are using HTCondor).

Reading Traces: Top-to-Bottom vs. Bottom-to-Top

Reading an error log sequentially from line 1 to line 200 is often a waste of time. Different programming languages and runtime tools print stack traces in entirely different directions.

Python: Read Bottom-to-Top

Python prints error traces chronologically, meaning the actual fatal exception is always printed at the very bottom of the trace.

• The Strategy: Scroll directly to the final line of the .err file to identify the exact error type (e.g., KeyError, IndexError, ModuleNotFoundError). Once you know what failed, scan upward to find the line pointing to your script rather than an internal library package.

C/C++ and Fortran: Read Top-to-Bottom

Compiled languages and build tools log errors chronologically as they are encountered during compilation or execution.

• The Strategy: Start at the very top of the error log. The first error printed is almost always the true root cause; the hundreds of lines following it are usually cascading failures triggered by that initial crash.

Bash / Shell Scripts: Read Where It Stops

By default, Bash scripts continue executing subsequent commands even if an earlier line fails (unless set -e is enabled).

• The Strategy: Search for the specific command line that failed by scanning upward from where output stopped matching your expected workflow.

Decoding the Usual Suspects: Classic HPC Failures

Most cluster job crashes stem from three recurring issues. Recognizing how each scheduler reports these failures allows you to fix them instantly.

1. Out Of Memory (OOM) / Exit Code 137 / Job Holds

• The Cause: Your program tried to consume more RAM than requested. To prevent your job from crashing neighboring processes on a shared node, the system kernel or scheduler terminated your process.
• How Schedulers Report It:
• SLURM: Your .err file or sacct -j <job_id> reports exit code 137 (128 + Signal 9 SIGKILL), often accompanied by slurmstepd: error: Detected allocation failure. OOM Killer invoked.
• UGE / SGE: The job fails with exit code 137. Running qacct -j <job_id> or qstat -j <job_id> shows failed: 100 : Assumed OS problem or reveals that the job exceeded its specified maxvmem ceiling.

• HTCondor: Rather than exiting outright, HTCondor usually places the job in a HOLD state (H). Running condor_q -hold or checking the .log file explicitly states: Job used more memory than requested.

• The Fix: Increase the requested memory in your submission script:
• SLURM: #SBATCH --mem=32G (or #SBATCH --mem-per-cpu=8G)
• UGE: #$ -l h_vmem=32G
• HTCondor: request_memory = 32GB

2. “No Space Left on Device” or Quota Exceeded

• The Symptom: IOError: [Errno 28] No space left on device or write error: disk quota exceeded.
• What it means: Your script ran out of disk space. This rarely happens on large global storage filesystems (/scratch or /project), but frequently occurs when temporary files auto-write to small localized /tmp partitions or constrained home directories (/home/username).
• The Fix: Redirect temporary directory environment variables to high-capacity scratch space directly inside your job script before running code:

export TMPDIR=/scratch/username/tmp
export PIP_CACHE_DIR=/scratch/username/pip_cache
mkdir -p $TMPDIR $PIP_CACHE_DIR

(In HTCondor, you can also request dedicated local scratch disk space via request_disk = 50GB in your submit file).

3. “Command Not Found” or “ModuleNotFoundError”

• The Symptom: bash: line 14: gfortran: command not found or ModuleNotFoundError: No module named 'torch'.
• What it means: Compute nodes start in a clean, isolated environment. They do not automatically inherit loaded modules, exported $PATH variables, or active Conda environments from your login session terminal.
• The Fix: Explicitly load system modules and activate virtual environments directly inside your execution script:

# Load system modules
module load python/3.11

# Activate your custom environment
source /scratch/username/envs/my_env/bin/activate

# Execute your application
python3 my_script.py

Quick Scheduler Diagnostic Cheat Sheet

When a job finishes or fails unexpectedly, use these diagnostic commands to query job metadata and exit status:

The Log Parsing Checklist

Before reaching out to cluster administrators or teammates for help with a broken job, run through these steps:

1. Locate the standard error log (.err). For HTCondor, inspect the workflow .log file as well for scheduler-level events.
2. Determine the language direction. Read Python errors from the bottom up; read C/C++ build failures or shell errors from the top down.
3. Use scheduler diagnostics. Check sacct (SLURM), qacct (UGE), or condor_q -hold (HTCondor) to verify exit codes and memory usage.
4. Identify resource limits. An exit status of 137 or a memory hold reason means your job needs a higher memory allocation.
5. Set explicit pathing. Ensure all module load commands, TMPDIR paths, and environment activations are written explicitly inside your job submission file.

Developing code locally inside a Jupyter Notebook or stepping through lines of Python with VS Code’s debugger is a fast, intuitive way to work. But when your local machine runs out of RAM, CPU cores, or GPU memory, you pack up your workflow and move to a High-Performance Computing (HPC) cluster.

Once there, many researchers hit an immediate friction point: clusters are traditionally built around non-interactive batch scripts submitted to job schedulers like SLURM or Grid Engine. Switching from an interactive graphical editor back to pure command-line scripts and print-statement debugging can feel like stepping backward in time.

Fortunately, you don’t have to give up your interactive tools to use a supercomputer. You just need to know how to bridge local GUIs with remote cluster compute—without breaking cluster rules.

The Cardinal Sin: Running Servers on Shared Login Nodes

When you SSH into a cluster, you land on a login node (or head node). Think of the login node as a shared front porch. It is meant exclusively for light administrative tasks: editing text files, organizing directory trees, checking queue status, and submitting job scripts.

The most common mistake new cluster users make is running jupyter notebook or connecting VS Code’s Remote-SSH plugin directly to the login node.

Why this breaks the cluster:

• VS Code Server (.vscode-server) spins up background Node.js processes, language servers, file watchers, and extension host daemons that constantly index workspace directories.
• Jupyter Notebooks execute heavy data manipulation, model training, and plotting live inside the server process.

Login nodes are shared simultaneously by hundreds of users. When an interactive server process starts hogging gigabytes of RAM and maxing out CPU cores, the entire command-line interface bogs down for everyone else. To prevent system crashes, administrators install automated automated watcher scripts that will abruptly kill any long-running or high-memory user process on a login node.

To run these tools properly, your server processes must run on a compute node.

Strategy 1: SSH Port Forwarding (ssh -L)

If you want to run Jupyter or VS Code manually, you can use SSH Local Port Forwarding (ssh -L). This technique creates a secure tunnel between a port on your local laptop, through the login node, and directly into an active compute node.

Step 1: Request an Interactive Compute Session

Before launching any server, request a temporary allocation on a compute node using your cluster’s scheduler:

# Request 2 CPU cores, 16GB RAM, and 2 hours on SLURM
srun --tasks=1 --cpus-per-task=2 --mem=16G --time=02:00:00 --pty bash

Once your shell transfers you to a compute node, take note of the node’s hostname (e.g., node042.cluster.internal).

Step 2: Launch the Server on the Compute Node

Start your Jupyter server on the compute node, making sure to disable automatic browser opening:

# Load your Python module or activate your Conda environment first
module load python
jupyter notebook --no-browser --port=8888

Terminal output will provide a URL containing a security token (e.g., http://localhost:8888/?token=a1b2c3d4...).

Step 3: Establish the SSH Tunnel from Your Local Laptop

Open a new terminal tab on your local laptop (not on the cluster) and set up the tunnel using ssh -L:

# Syntax: ssh -L [Local_Port]:[Compute_Node_Hostname]:[Remote_Port] [Username]@[Login_Node]
ssh -L 8888:node042.cluster.internal:8888 netid@cluster.university.edu

This command forwards traffic from your laptop’s local localhost:8888 through the login node jump host directly to port 8888 on compute node042.

Step 4: Access the Interface

Open your local web browser and navigate to http://localhost:8888. Paste the security token generated in Step 2, and you now have a full Jupyter GUI running on dedicated cluster hardware!

VS Code Remote Tip: When using VS Code’s Remote SSH extension on shared systems, home directory file locking can sometimes break connections. Refer to resources like Notre Dame’s CRC VS Code Documentation for cluster-specific settings, such as enabling “Lockfiles In Tmp” in your extension settings to ensure seamless reconnects.

Strategy 2: Open OnDemand (The Modern Web Portal)

Setting up SSH tunnels manually works well, but it requires managing port numbers, tokens, and multiple terminal windows. Many modern HPC centers simplify this entirely by deploying Open OnDemand (OOD).

Open OnDemand is an open-source, web-based portal that gives you graphical access to cluster resources straight through a web browser without needing any local SSH configuration or command-line tunneling.

How Open OnDemand Works:

1. Log in via Web Browser: Navigate to your university’s Open OnDemand URL (e.g., ondemand.crc.nd.edu) and authenticate with your institutional login.
2. Select an Interactive App: Choose Jupyter, VS Code, RStudio, or MATLAB from the interactive applications menu.
3. Configure Your Resource Allocation: A form will ask you to specify your required resources (number of CPUs, GPUs, RAM amount, and maximum runtime).
4. Launch: Click Launch.

Under the hood, Open OnDemand automatically writes a job script, submits it to the cluster scheduler (SLURM/Grid Engine), waits for a compute node to be allocated, launches the server process, builds a secure web proxy, and presents a “Connect” button in your browser window.

Because Open OnDemand routes all compute through scheduled jobs, you get the full speed of dedicated compute hardware and GPUs without violating cluster usage policies or risking process kills on login nodes.

Summary: The Interactive Computing Checklist

Interactive development and HPC compute can easily coexist if you follow these rules:

1. Never run Jupyter Notebooks or VS Code Server processes on login nodes. Keep login nodes clean for light editing and job submission.
2. Check for Open OnDemand first. If your site hosts an Open OnDemand portal, use it for one-click interactive web sessions.
3. Allocate compute nodes before launching servers manually. Always start with an interactive job allocation (srun or qsub).
4. Tunnel securely with ssh -L. Forward local ports through the login host to the specific compute node hosting your active server.
5. Clean up when finished. Terminate your interactive compute sessions and close notebook servers when you finish working so cluster resources return to the queue for other researchers.