All in One View

Content from Introduction to SLURM

Last updated on 2026-08-27 | Edit this page

Estimated time: 20 minutes

Overview

Questions

  • What is a job scheduler and why do we need one?
  • What is SLURM?
  • Why can’t I just run jobs on the head node?

Objectives

  • Understand the purpose of job schedulers in shared computing environments
  • Learn the basic architecture of Sagehen HPC
  • Distinguish between head nodes and compute nodes
  • Understand the SLURM job lifecycle

What is a Job Scheduler?

When multiple researchers share a computing cluster, a job scheduler coordinates access:

  1. Manages resources – tracks CPU, memory, and GPU availability
  2. Queues jobs – accepts requests and holds them until resources are free
  3. Allocates resources – assigns specific compute nodes to jobs
  4. Ensures fairness – prevents any one user from monopolizing the cluster
  5. Enforces policies – implements time limits, quotas, and priority rules

Without a scheduler, researchers would need to manually coordinate who runs what and when.

SLURM: Simple Linux Utility for Resource Management

SLURM is the job scheduler used on Sagehen. It is one of the most widely deployed schedulers in HPC, running on thousands of clusters worldwide.

Key SLURM Concepts

  • Jobs: A unit of work – either interactive (real-time shell) or batch (scripted, unattended)
  • Partitions: Logical groupings of nodes with different characteristics
  • Resources: What you request for your job (CPU cores, memory, time, GPUs)

The Sagehen HPC Cluster Architecture

Head Node

  • Address: sagehen.hpc.pomona.edu
  • CPU: 2 threads; RAM: 8 GB
  • Purpose: Login, submit jobs, manage files
Diagram of Sagehen HPC. Your laptop connects by SSH to the head node at sagehen.hpc.pomona.edu, which is for login and job submission only. From there SLURM dispatches work to three partitions: amd, the default, with 12 nodes a001 to a012 of 128 cores and 500 GB each and up to 720 hours; gpu, with 10 GPUs in total made up of four A100 80GB, four L40S 48GB and two RTX PRO 6000 96GB; and short, which shares the amd nodes, is capped at 2 hours and starts almost immediately. Below, shared storage visible from every node: /rhome with 100 GB per user backed up, /bigdata/lab per lab with 1 TB backed up, and /scratch node-local SSD deleted at job end.
The head node takes your submission; SLURM decides which compute node actually runs it.
Callout

The Head Node is Sacred

The head node has only 2 CPU threads and 8 GB of RAM. Running compute jobs there will crash the login server, kill other users’ jobs, and result in account restrictions. Always submit jobs to compute nodes via SLURM.

Compute Nodes

Partition Specs Max Time Best For
amd See sinfo -p amd for current configuration 30 days General compute
gpu 10 GPUs across multiple nodes (4× A100, 4× L40S, 2× RTX PRO 6000; see Workshop 16) 30 days ML, GPU-accelerated work
short See sinfo -p short for current configuration Shorter max walltime than amd/gpu Quick test jobs, debugging, rapid prototyping
A job submitted with sbatch enters the PENDING state while it waits for free CPUs or a GPU, for your other jobs to finish, or for the partition limits to allow it. It then moves to RUNNING while the script executes on a compute node. From there it ends as COMPLETED with exit code zero, FAILED with a non-zero exit, or CANCELLED by scancel or a timeout. All three can be checked with sacct -j jobid.
Every job ends in one of three states, and sacct will tell you which.

The SLURM Job Lifecycle

  1. Submission: You run sbatch (batch) or srun (interactive) on the head node
  2. Queue: SLURM accepts your job and places it in a queue
  3. Waiting: SLURM waits for available resources and priority scheduling
  4. Allocation: SLURM assigns specific compute nodes
  5. Execution: Your job runs on those nodes
  6. Completion: Output is written to files; nodes are freed

SLURM Command Overview

Command Purpose
srun Start an interactive job
sbatch Submit a batch script
squeue View jobs in the queue
scancel Cancel a job
sacct View job history
seff Analyze job efficiency
sinfo View cluster status
Challenge

Challenge 1: Exploring Sagehen HPC

Connect to Sagehen and explore its structure.

  1. Connect: ssh <myusername>@sagehen.hpc.pomona.edu

  2. Complete DUO MFA authentication

  3. Check the head node:

    BASH 

    hostname
nproc
free -h

  4. View partitions:

    BASH 

    sinfo
sinfo -s

Look for the amd, gpu, and short partitions in the output.

You should see sagehen from hostname, 2 from nproc, and ~8 GB from free -h. The sinfo output shows the amd, gpu, and short partitions with their node counts and time limits.

Key Points
  • A job scheduler coordinates resource sharing on a cluster
  • SLURM is the job scheduler on Sagehen
  • The head node (2 CPUs, 8 GB RAM) is for login and job submission only
  • Jobs run on compute nodes in three partitions: amd (general compute), gpu (GPU-accelerated), and short (quick test/debug jobs)
  • SLURM enforces fair access through queuing, resource limits, and priorities

Content from Partitions and Resources

Last updated on 2026-08-27 | Edit this page

Estimated time: 25 minutes

Overview

Questions

  • What partitions are available on Sagehen HPC?
  • How do I choose the right partition for my job?
  • What GPU types are available?

Objectives

  • Understand the characteristics and use cases for each Sagehen partition
  • Match job requirements to the appropriate partition
  • Know the available GPU types and their strengths

The “amd” Partition (General Compute)

The workhorse of Sagehen, suitable for most research computing.

For the current node count, CPU count, and RAM configuration, run sinfo -p amd on Sagehen.

Property Value
Configuration See sinfo -p amd for current details
Max job time 720 hours (30 days)

Use cases: Statistical analysis, bioinformatics, physics simulations, data processing, batch processing.

The “gpu” Partition (GPU Acceleration)

For workloads requiring GPU acceleration.

Property Value
Nodes Multiple GPU-equipped nodes
CPUs per node 128 cores, 500 GB RAM
Max job time 720 hours (30 days)

Available GPU Types

Sagehen has 10 GPUs total (confirmed May 2026):

GPU Type Memory Quantity Best For
NVIDIA A100 80 GB HBM2e 4 Large models, production AI
NVIDIA L40S 48 GB GDDR6 4 Deep learning, inference
NVIDIA RTX PRO 6000 96 GB GDDR7 ECC 2 Largest memory on the cluster

Check availability: sinfo -p gpu --Format=NodeList,Gres,GresUsed

Callout

Share GPUs Responsibly

GPU nodes are expensive and limited: - Only request GPUs if your code actually uses them - Use L40S for prototyping, A100 for production - Monitor GPU utilization with nvidia-smi - Request the minimum number you need

The “short” Partition (Quick Jobs and Debugging)

For short-duration test jobs, debugging, and rapid prototyping. The short partition has a shorter maximum walltime than amd and gpu, but jobs typically start more quickly because the scheduler can back-fill them between longer jobs.

Property Value
Configuration See sinfo -p short for current details
Max job time Shorter than amd/gpu — verify with sinfo -p short

Use cases: Quick test of a new submission script, single-task debugging, short interactive sessions, sanity-checking a workflow before launching a 30-day production run.

Example: a quick test job on the short partition

BASH 

#!/bin/bash
#SBATCH --job-name=script_test
#SBATCH --partition=short
#SBATCH --time=00:15:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --output=test_%j.log

module purge
module load miniconda3

python my_script.py --dry-run

If the test succeeds, you can change --partition=short to --partition=amd (and increase --time) for the production run.

A decision tree for choosing a partition. If you need a GPU, use the gpu partition with four A100 80GB, four L40S 48GB and two RTX PRO 6000 96GB cards, allowing up to 720 hours. If not, ask whether the job will finish in under two hours; if it will, use short, which shares the amd nodes and starts almost immediately. Otherwise use amd, the default, with 12 nodes of 128 cores and 500 GB each and up to 720 hours.
Two questions settle it: do you need a GPU, and will the job finish inside two hours?

Choosing the Right Partition

  1. Need GPU acceleration? -> gpu partition
  2. Quick test, debug, or short prototype run (within short walltime limit)? -> short partition
  3. Everything else (production CPU work) -> amd partition

Account and Group Limits

Each account has resource limits to ensure fair sharing:

Limit Details
Max cores per account Per-account limits apply
Max GPUs per account Per-account limits apply
Max submitted jobs Per-account limits apply

Check your group’s quota with quota_check.sh. Contact for current limits or temporary increases.

Challenge

Challenge 1: Matching Jobs to Partitions

For each scenario, decide which partition to use:

  1. Testing a new analysis script (takes about 30 seconds)
  2. Analyzing a 1 TB dataset, expected to take 6 hours
  3. Training a deep learning model, expected to take 18 hours
  4. Running a molecular dynamics simulation for 72 hours
  1. short – Quick test (~30 seconds) is the canonical use case for the short partition
  2. amd – General compute, no GPU needed; 6 hours fits within amd’s 30-day limit
  3. gpu – Deep learning needs GPU acceleration
  4. amd – Long simulation (72 hours), no GPU needed, fits 30-day amd limit
Key Points
  • Sagehen has three partitions: amd (general compute), gpu (accelerators), and short (quick / test / debug jobs)
  • Choose partition based on GPU need and job duration
  • GPU partition offers A100 (production), L40S (inference), and RTX PRO 6000 (large-memory training)
  • Per-account resource limits apply; check with quota_check.sh
  • amd and gpu support up to 720 hours (30 days); short has a shorter walltime — check sinfo -p short

Content from Requesting Resources

Last updated on 2026-08-27 | Edit this page

Estimated time: 25 minutes

Overview

Questions

  • How do I specify CPU, memory, time, and GPU resources in SLURM?
  • How do I estimate what resources my job needs?
  • What storage options are available during a job?

Objectives

  • Master SLURM resource request syntax for CPU, memory, time, and GPUs
  • Learn how to estimate resource needs for different workloads
  • Understand storage options available during jobs

SLURM Resource Request Keywords

CPU and Memory

Parameter Meaning Example
-c / --cpus-per-task CPU cores per task -c 4
-N / --nodes Number of nodes -N 1
--mem RAM per node --mem=50G
--mem-per-cpu RAM per CPU core --mem-per-cpu=4G

Time

Parameter Example Meaning
-t / --time -t 00:30:00 30 minutes
-t 1-12:00:00 1 day 12 hours
-t 1440 1440 minutes (1 day)

GPUs

Parameter Example Meaning
--gres=gpu:N --gres=gpu:2 2 GPUs (any type)
--gres=gpu:type:N --gres=gpu:a100:1 1 A100 specifically

Partition

Parameter Example
-p / --partition -p amd
Four SLURM flags and guidance for each. --cpus-per-task: start at 1 and only raise it if your code actually uses more than one core. --mem: ask for a little more than your peak and check the real figure with seff. --time: your estimate plus about 25 percent, since too short kills the job and far too long delays scheduling. --gres=gpu:1: only on the gpu partition, naming the type if you need one such as gpu:a100:1 or gpu:l40s:1. A closing note warns that asking for far more than you use means waiting longer in the queue for no benefit.
Four flags, and what each one costs you if you get it wrong.

Estimating Resource Needs

CPU Cores

  1. Is your code multi-threaded? Check if it uses OpenMP, multiprocessing, etc.
  2. How many threads does it spawn?
  3. Rule of thumb: Start with 4 cores. Double if too slow.

Memory

  1. How big are your input datasets?
  2. Does the algorithm create temporary copies?
  3. Rule of thumb: Request 2-4x the size of your largest input file.

Time

  1. Have you run this job before? How long did it take?
  2. Rule of thumb: Request 2x your best guess. SLURM runs your job as soon as there’s space.

GPUs

  1. Does your code use CUDA, PyTorch, or TensorFlow?
  2. Rule of thumb: Start with 1 GPU. Check utilization with nvidia-smi.

Common Resource Request Patterns

Small CPU job (quick test)

BASH 

srun -c 2 --mem=8G -t 00:30:00 -p short my_script.py

Medium CPU job

BASH 

srun -c 16 --mem=128G -t 04:00:00 -p amd my_analysis.R

Large CPU job

BASH 

srun -c 64 --mem=450G -t 24:00:00 -p amd heavy_simulation.sh

Single GPU job

BASH 

srun --gres=gpu:a100:1 -c 8 --mem=64G -t 12:00:00 -p gpu train_model.py

Storage During Jobs

Storage Speed Persistence Use For
/rhome Medium Persistent Source code, results
/bigdata Medium Persistent Shared datasets
/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID Fast (SSD) Deleted when job completes Working files
/tmpfs Very fast (RAM) Deleted when job ends Hot data
Challenge

Challenge 1: Resource Estimation

Estimate the resources needed for these jobs:

  1. Processing a 10 GB CSV file with pandas (under 1 hour)
  2. Training a ResNet-50 model on ImageNet (24 hours on a single GPU)
  3. Compiling a large C++ codebase with 16 parallel jobs
  1. 10 GB CSV: -c 4 --mem=50G -t 01:00:00 -p amd (4 cores, 5x file size for memory)
  2. ResNet-50: --gres=gpu:a100:1 -c 8 --mem=64G -t 24:00:00 -p gpu
  3. C++ compilation: -c 16 --mem=32G -t 00:30:00 -p short (parallel make with -j16; short walltime fits the short partition)

Start conservative and adjust based on actual usage from seff.

Key Points
  • Resource requests use SLURM flags: -c (cores), --mem (memory), -t (time), --gres=gpu (GPUs)
  • Start conservative with estimates; monitor actual usage with seff and adjust
  • Time format: HH:MM:SS or D-HH:MM:SS for longer jobs
  • Use /scratch for fast temporary I/O during jobs; use /rhome or /bigdata for persistent results
  • Over-requesting wastes resources and makes other users wait longer

Content from Interactive Jobs

Last updated on 2026-08-24 | Edit this page

Estimated time: 30 minutes

Overview

Questions

  • How do I get an interactive shell on a compute node?
  • How do I request resources for an interactive session?
  • When should I use interactive vs batch jobs?

Objectives

  • Master srun --pty bash for interactive sessions
  • Request CPU, memory, and GPU resources for interactive jobs
  • Use interactive sessions for development and debugging
  • Know when to switch to batch jobs

The srun Command

srun starts interactive jobs. The most common form for a short interactive debugging session:

BASH 

srun -c 4 --mem=16G -t 00:30:00 -p short --pty bash -l
  • -c 4: 4 CPU cores
  • --mem=16G: 16 GB RAM
  • -t 00:30:00: 30 minutes
  • -p short: short partition (good for short interactive work; for sessions longer than the short walltime limit, use -p amd)
  • --pty bash -l: interactive login shell

After SLURM allocates resources, your prompt changes to show the compute node:

[user@compute-node ~]$

Type exit or Ctrl+D to end the session and free resources.

Common Interactive Patterns

Quick Test (2 minutes, 1 core)

BASH 

srun -c 1 --mem=4G -t 00:02:00 -p short --pty bash -l

Development Session (30 minutes, 4 cores)

BASH 

srun -c 4 --mem=16G -t 00:30:00 -p short --pty bash -l

Large Interactive (2 hours, 16 cores)

BASH 

srun -c 16 --mem=100G -t 02:00:00 -p amd --pty bash -l

(For interactive work longer than the short partition’s max walltime, use -p amd instead.)

Interactive GPU Session

BASH 

srun --gres=gpu:a100:1 -c 8 --mem=64G -t 04:00:00 -p gpu --pty bash -l
Callout

Request Time Generously

If you finish early, just exit to free resources. If time runs out, your session is killed abruptly. Better to release early than get a timeout surprise.

Working in Interactive Sessions

Load modules and run code just like on your laptop:

BASH 

[user@compute-node ~]$ module load miniconda3
[user@compute-node ~]$ python my_analysis.py

Edit and rerun for iterative development:

BASH 

[user@compute-node ~]$ nano my_script.R
[user@compute-node ~]$ Rscript my_script.R

Using tmux for Persistent Sessions

If your connection drops, a regular interactive session dies. Use tmux inside your session:

BASH 

tmux new-session -s work
# Detach: Ctrl+B, then D
# Reconnect: tmux attach-session -t work

When to Use Interactive vs Batch

Interactive: developing code, debugging, exploring data, quick tests

Batch: stable code, jobs over 2 hours, many parallel jobs, unattended runs

Challenge

Challenge 1: Your First Interactive Session

  1. Start a session: srun -c 2 --mem=8G -t 00:10:00 -p short --pty bash -l

  2. Verify you are on a compute node:

    BASH 

    hostname
echo $SLURM_JOB_ID
free -h
nproc

  3. Load Python and run a quick test:

    BASH 

    module load miniconda3
python -c "import numpy as np; print(np.random.randn(5))"

  4. Exit: exit

  5. Confirm you are back on the head node: hostname

Inside the session, hostname shows a compute node (e.g., a005), free -h shows 500 GB total RAM, and nproc shows 128 cores. After exit, hostname returns sagehen. The Python test confirms modules work on compute nodes.

Key Points
  • Use srun --pty bash -l for interactive sessions on compute nodes
  • Request resources: -c (cores), --mem (RAM), -t (time), -p (partition)
  • Interactive jobs are best for development, testing, and debugging
  • Use the short partition for short interactive sessions and quick debugging; use amd for longer interactive work
  • Use tmux inside sessions for connection resilience
  • Switch to batch jobs when code is stable and needs long or unattended runs

Content from Writing Batch Scripts

Last updated on 2026-08-24 | Edit this page

Estimated time: 25 minutes

Overview

Questions

  • What is a batch script and how is it structured?
  • What are the essential SBATCH directives?
  • How do I load modules and run code in batch scripts?

Objectives

  • Write complete SLURM batch scripts
  • Use SBATCH directives to request resources
  • Load modules and configure environments in scripts
  • Understand SLURM environment variables available in jobs

Why Batch Jobs?

Batch jobs run unattended while you do other things. They are the backbone of research computing:

  • Run long jobs (hours, days, weeks)
  • Submit many jobs at once
  • Reproducible: exact commands saved in the script
  • Run overnight or over weekends
  • Get email notifications when done

Anatomy of a Batch Script

BASH 

#!/bin/bash -l
#SBATCH --job-name=my_job
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=00:30:00
#SBATCH --partition=amd
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err

module load miniconda3

python my_analysis.py

The Shebang Line

BASH 

#!/bin/bash -l

Always use -l so your .bashrc is read and modules are available.

SBATCH Directives

Directive Meaning Example
--job-name Name shown in squeue --job-name=analysis_v1
--cpus-per-task CPU cores --cpus-per-task=8
--mem Total RAM --mem=32G
--time Max job time --time=04:00:00
--partition Partition --partition=amd
--output stdout file (%j = job ID) --output=job_%j.out
--error stderr file --error=job_%j.err
--gres GPUs --gres=gpu:2
--mail-user Email address --mail-user=me@pomona.edu
--mail-type When to email --mail-type=END,FAIL

SLURM Environment Variables

SLURM sets useful variables inside your job:

Variable Meaning
$SLURM_JOB_ID Job ID
$SLURM_JOB_NAME Job name
$SLURM_CPUS_PER_TASK Cores requested
$SLURM_ARRAY_TASK_ID Array task ID

Example Batch Scripts

Python Data Analysis

BASH 

#!/bin/bash -l
#SBATCH --job-name=data_analysis
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=01:00:00
#SBATCH --partition=amd
#SBATCH --output=analysis_%j.out

module load miniconda3
python analysis.py

R Statistical Analysis (short test run)

BASH 

#!/bin/bash -l
#SBATCH --job-name=stats
#SBATCH --cpus-per-task=2
#SBATCH --mem=8G
#SBATCH --time=00:30:00
#SBATCH --partition=short
#SBATCH --output=r_analysis_%j.out

module load r/4.5.1
Rscript analysis.R

Quick R analyses with a short walltime are a good fit for the short partition. For longer R jobs, change --partition=short to --partition=amd.

GPU Deep Learning

BASH 

#!/bin/bash -l
#SBATCH --job-name=train_model
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=12:00:00
#SBATCH --partition=gpu
#SBATCH --gres=gpu:a100:1
#SBATCH --output=train_%j.out

module load cuda miniconda3
source ~/ml_env/bin/activate
python train.py --epochs 100

Best Practices

Stop on First Error

BASH 

#!/bin/bash -l
set -e    # Exit on first error

python step1.py  # Must succeed
python step2.py  # Only runs if step1 succeeds

Include Logging

BASH 

echo "Job started at $(date)"
echo "Running on $(hostname)"
echo "Job ID: $SLURM_JOB_ID"

python my_analysis.py

echo "Job completed at $(date)"

Use Scratch for I/O

BASH 

cp /rhome/$USER/large_input.dat /scratch/$SLURM_JOB_USER/$SLURM_JOB_ID/
cd /scratch/$SLURM_JOB_USER/$SLURM_JOB_ID/
python process.py
cp output.csv /rhome/$USER/results/
Challenge

Challenge 1: Write a Batch Script

Create simple_job.sh that:

  1. Uses the short partition with 1 core and 1 GB RAM for 5 minutes
  2. Prints the hostname, job ID, and current date
  3. Sleeps for 10 seconds
  4. Prints “Job complete!”

BASH 

#!/bin/bash -l
#SBATCH --job-name=hello_world
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=00:05:00
#SBATCH --partition=short
#SBATCH --output=hello_%j.out

echo "Hello from Sagehen!"
echo "Job ID: $SLURM_JOB_ID"
echo "Running on: $(hostname)"
sleep 10
echo "Job complete!"

Save this file and make it executable with chmod +x simple_job.sh.

The script includes all required SBATCH directives and uses SLURM environment variables. The %j in the output filename ensures each run creates a unique file. The sleep 10 simulates actual computation.

Key Points
  • Batch scripts are bash scripts with #SBATCH directives for resource requests
  • First line must be #!/bin/bash -l (the -l flag loads your environment)
  • Key directives: --job-name, --cpus-per-task, --mem, --time, --partition, --output
  • Always load modules inside the script, not just interactively
  • Use set -e to stop on first error
  • Include logging (date, hostname, job ID) for debugging
  • Use scratch storage for I/O-intensive intermediate files

Content from Batch Job Submission

Last updated on 2026-08-27 | Edit this page

Estimated time: 35 minutes

Overview

Questions

  • How do I submit a batch job?
  • How do I check job status and progress?
  • Where are my job results, and how do I follow them in real time?
  • How do I get email notifications without spamming myself?

Objectives

  • Submit jobs with sbatch and verify submission
  • Read SLURM status codes and find the cause of pending jobs
  • Locate and view output files during and after a run
  • Configure email notifications appropriately for the job size
  • Recognize and avoid the most common submission mistakes

Why Batch Submission Is the Right Default

The shell prompt on the Sagehen HPC login node is convenient but not where your real work belongs. Login nodes are shared and intentionally constrained: a long calculation on the login node steals resources from every other user trying to edit, transfer, or schedule. Batch submission moves your work to the compute partition where SLURM can give it dedicated CPU, memory, and time, and where your job survives if you close your laptop.

The pattern is consistent: you write a short shell script that includes both SLURM directives (lines starting with #SBATCH) and the commands to run, then submit it with sbatch. SLURM puts the job in queue, runs it when resources are available, and writes its stdout and stderr to files in your current directory.

Submitting with sbatch

BASH 

sbatch my_script.sh

SLURM responds immediately:

Submitted batch job 12345

Your job ID is 12345. Use this number to track and manage the job. The submitting shell can close, your laptop can sleep, you can disconnect from the VPN, and the job continues to run.

Checking Job Status

BASH 

# All your jobs
squeue -u $USER

# A specific job
squeue -j 12345

# Estimated start time for pending jobs
squeue -u $USER --start

Output:

     JOBID PARTITION     NAME     USER ST       TIME  NODES CPUS
     12345       amd analysis   alice  R       0:23      1    4

Status codes:

  • R Running: actively executing
  • PD Pending: waiting for resources, see the REASON field for why
  • CG Completing: cleanup phase, almost done
  • CD Completed: finished successfully
  • F Failed: ended with non-zero exit code
  • CA Cancelled: stopped via scancel
  • TO Timeout: exceeded --time limit and was killed
Callout

Decoding “why is my job pending?”

squeue -u $USER shows a (REASON) column for pending jobs:

(Resources): waiting for matching nodes to become available. Normal for busy partitions; just wait.

(Priority): jobs with higher priority are ahead. This is fair-share doing its job.

(QOSMaxJobsPerUserLimit): you have hit a per-user concurrency cap. Wait for one of yours to finish.

(ReqNodeNotAvail): the constraint you set (e.g., a specific GPU type) currently has no available node. May take a long time.

(Dependency): job is waiting for another job to complete (--dependency=afterok:JOBID).

Output Files

By default, SLURM writes a single combined log to slurm-JOBID.out in the directory where you submitted. For most workflows, separating stdout and stderr is more useful:

BASH 

#SBATCH --output=analysis_%j.out   # stdout
#SBATCH --error=analysis_%j.err    # stderr

The %j placeholder is replaced by the job ID. Other useful placeholders: %x for job name, %a for array task ID, %N for node name. A common pattern that puts logs in their own folder:

BASH 

#SBATCH --output=logs/%x_%j.out
#SBATCH --error=logs/%x_%j.err

Make sure logs/ exists before submitting. SLURM will fail to start the job if the output path is invalid.

Viewing Output

BASH 

# While running (real-time)
tail -f analysis_12345.out

# After completion
cat analysis_12345.out

# Last 50 lines (useful for long jobs)
tail -50 analysis_12345.out

# Search for a specific string
grep -i error analysis_12345.out

You can also specify a full path so logs land somewhere predictable:

BASH 

#SBATCH --output=/rhome/alice/logs/job_%j.out

Email Notifications

BASH 

#SBATCH --mail-user=alice@pomona.edu
#SBATCH --mail-type=END        # When finished
#SBATCH --mail-type=FAIL       # On failure
#SBATCH --mail-type=ALL        # Any state change (verbose)
Callout

Email Best Practices

For testing and short jobs: no email at all. Watch with squeue instead.

For medium jobs (hours): FAIL only. You will be notified about problems but not flooded with completion notices.

For long important jobs (days): END,FAIL. You want to know either way, but only once.

For job arrays: FAIL only. Otherwise a 100-task array sends 100 completion emails.

The BEGIN mail-type exists but is almost never useful: by the time the email arrives, your job has already started.

Changing Directory

By default, scripts run in the directory where you submitted. If your submitted script is in /rhome/alice/scripts/ but your data is in /rhome/alice/project/, the script’s relative paths will break.

Two solutions, in order of preference:

BASH 

# Option A: cd at the top of the script (clearest)
cd /rhome/alice/project
python analysis.py

# Option B: use absolute paths everywhere (works but verbose)
python /rhome/alice/scripts/analysis.py /rhome/alice/project/data.csv

For jobs that read or write a lot, cd to a directory under /scratch instead of /rhome (covered in Workshop 12).

Common Submission Mistakes

A few mistakes show up over and over in real submissions:

Forgetting --time. The default is short. A long job submitted without an explicit time limit will be killed mid-run.

Setting --mem to the maximum because you do not know what the job needs. This makes the job harder to schedule. Run a short test first with seff JOBID to right-size memory.

Mixing tabs and spaces in the SLURM directives. Some directives parse silently incorrectly. Stick to spaces.

Forgetting to load required modules inside the script. The login-node environment does not carry over. Always include module load miniconda3 (or whatever you need) at the top of the script.

Submitting from /scratch. Output files go to the submission directory. Submit from /rhome so logs persist after the job’s scratch is cleaned.

Challenge

Challenge 1: Submit and Monitor a Batch Job

  1. Create analysis.py:

    PYTHON 

    import time
print("Starting analysis...")
start = time.time()
result = sum(range(10000000))
print(f"Result: {result}")
print(f"Completed in {time.time() - start:.2f}s")

  2. Create run_analysis.sh:

    BASH 

    #!/bin/bash -l
#SBATCH --job-name=python_test
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --time=00:05:00
#SBATCH --partition=short
#SBATCH --output=analysis_%j.out

module load miniconda3
python analysis.py

  3. Submit: sbatch run_analysis.sh

  4. Check status: squeue -u $USER

  5. View output: cat analysis_JOBID.out

  6. After completion, run seff JOBID to see actual CPU and memory usage.

After submission, squeue shows the job briefly. The output file contains the computation result and timing. The seff JOBID output reveals you over-requested memory: a sum of 10 million integers needs about 50 MB, not 2 GB. On the next run, drop --mem to 512M. This iteration is exactly how you tune SLURM scripts: submit, watch, measure with seff, adjust.

Challenge

Challenge 2: Read a Pending Job

A friend submitted a job 30 minutes ago and it is still pending. They run squeue -u $USER and see:

JOBID PARTITION     NAME      USER ST       TIME  NODES NODELIST(REASON)
54321       gpu  big_train   bob   PD       0:00      1 (Resources)

Explain in plain English what the REASON column means and one thing they could do.

(Resources) means the GPU partition has no node currently available that matches the job’s requirements. SLURM is just waiting for a GPU node to free up.

Two reasonable responses: wait it out (queues clear naturally), or run squeue -p gpu --start to see estimated start times for all pending GPU jobs and decide if it is worth waiting. If the job needs an A100 specifically, switching to --gres=gpu:l40s:1 or --gres=gpu:l40s:1 will likely start sooner because those GPUs are less oversubscribed.

Key Points
  • Submit batch jobs with sbatch script.sh; the returned ID is how you track everything later
  • Status codes: R (running), PD (pending), CG (completing), CD (done), F (failed), TO (timeout)
  • The (REASON) column for pending jobs tells you why they are not running yet
  • Output files use --output and --error directives; %j and %x substitute job ID and name
  • Use tail -f to watch output in real-time while a job runs
  • Email: FAIL for medium jobs, END,FAIL for long ones, nothing for tests, FAIL only for arrays
  • Always run seff JOBID after completion to right-size future submissions
  • You can log out after submission; jobs run unattended

Content from Monitoring Running Jobs

Last updated on 2026-08-27 | Edit this page

Estimated time: 25 minutes

Overview

Questions

  • How do I monitor what my job is doing?
  • How do I check the history of completed jobs?
  • How efficient was my job? Did I waste resources?

Objectives

  • Master squeue for real-time job status
  • Use sacct to analyze completed job history
  • Use seff to evaluate job efficiency
  • Monitor node resources with free, top, and nvidia-smi

The Job Lifecycle

Every SLURM job moves through states:

SUBMITTED -> PENDING -> RUNNING -> COMPLETED/FAILED/CANCELLED

Viewing Jobs with squeue

Basic Usage

BASH 

squeue -u $USER              # All your jobs
squeue -j 12345              # Specific job
squeue --states=RUNNING      # Only running jobs
squeue --states=PENDING      # Only pending jobs
Terminal on Sagehen HPC showing squeue output for one user. Three jobs named demo1, demo2, and demo3 run on the short partition in state R for 25 seconds each, on nodes a001 and a002.
Three running jobs as squeue -u $USER shows them — state R, elapsed time, and the node each landed on.

Useful Filters

BASH 

# Jobs on a specific partition
squeue -p amd

# Custom output format
squeue -u $USER --format=jobid,name,state,elapsed,timelimit

# Watch in real-time (refreshes every 2 seconds)
watch -n 2 squeue -u $USER
A troubleshooting map starting from the statement that a job is not doing what you expect. Four branches: still PENDING, where squeue and the REASON column tell you whether it is waiting for resources or behind other jobs in priority; TIMEOUT, meaning the job hit its --time limit; OUT_OF_MEMORY, meaning --mem was too low and seff will show what it needed; and FAILED, where the error file is the place to start. A closing line gives the sacct command that reports all of these once the job has finished.
Start from the state SLURM reports, then work backwards.

Job History with sacct

After a job finishes, squeue no longer shows it. Use sacct:

BASH 

# Recent completed jobs
sacct -u $USER

# Detailed format
sacct -u $USER --format=jobid,jobname,state,elapsed,exitcode,maxrss

# Failed jobs only
sacct -u $USER --state=FAILED

# Jobs from a specific date
sacct -u $USER --starttime=2026-03-04
Terminal output of sacct filtered to the last five minutes. Jobs 38366 through 38369 show state COMPLETED with exit code 0:0, while job 38370 named fail shows FAILED with exit code 1:0. Each job lists batch and extern steps with MaxRSS values.
sacct history with exit codes: completed jobs show 0:0, the deliberately failing job shows FAILED 1:0 (this run added --starttime=now-5minutes to narrow the window).

Understanding Exit Codes

  • 0: Success
  • 1-127: Error from your program
  • 138: Timeout (exceeded time limit)
  • 143: Cancelled by user or system

Job Efficiency with seff

seff shows whether you wasted resources:

BASH 

seff 12345

Output:

Job ID: 12345
State: COMPLETED (exit code 0)
Cores: 4
Job Wall-clock time: 01:22:45
  Percent of CPUs in use: 100.0%
Memory Requested: 16.00 GB
Memory Used: 2.34 GB
  Percent of Memory in use: 14.6%

Interpreting: - CPU usage near 100% = good match - CPU usage below 50% = request fewer cores next time - Memory usage near 100% = request more next time - Memory usage below 20% = request less next time

Monitoring Nodes Directly

If you SSH to the compute node running your job:

BASH 

ssh a005           # Connect to the node
free -h            # Memory usage
top                # CPU usage (press q to exit)
nvidia-smi         # GPU usage (on GPU nodes)
logout             # Return to head node
Callout

Disk Usage on BeeGFS

Use quota_check.sh to check storage usage on Sagehen HPC’s BeeGFS filesystem. The standard du command does not work correctly for quota checks.

Challenge

Challenge 1: Monitor and Analyze a Job

  1. Submit a test job:

    BASH 

    sbatch --cpus-per-task=2 --mem=4G --time=00:05:00 -p short --wrap="sleep 30; echo Done"

  2. Monitor with: watch -n 1 squeue -u $USER

  3. After completion, check history: sacct -u $USER --format=jobid,state,exitcode | tail -5

  4. Analyze efficiency: seff JOBID

The job runs for 30 seconds then completes. sacct shows STATE=COMPLETED with exit code 0:0. seff shows low efficiency because we requested 2 cores but sleep only uses 1 thread. Real compute jobs should have much better efficiency.

Key Points
  • Use squeue for real-time job status and watch for continuous monitoring
  • Use sacct to view completed job history and exit codes
  • Use seff to analyze resource efficiency and adjust future requests
  • Exit code 0 = success, 138 = timeout, 143 = cancelled
  • Monitor nodes with free -h, top, and nvidia-smi
  • Use quota_check.sh for BeeGFS storage usage (not du)

Content from Managing and Canceling Jobs

Last updated on 2026-08-24 | Edit this page

Estimated time: 20 minutes

Overview

Questions

  • How do I cancel a running or queued job?
  • What do I do when a job is stuck pending?
  • How do I troubleshoot common job failures?

Objectives

  • Use scancel to cancel jobs safely
  • Diagnose why jobs are stuck in PENDING state
  • Troubleshoot timeout, out-of-memory, and other common failures

Canceling Jobs with scancel

Cancel a Single Job

BASH 

scancel 12345

The job stops immediately. Output files written so far are preserved.

Cancel Multiple Jobs

BASH 

# Cancel all your jobs
scancel -u $USER

# Cancel by name
scancel --name=test

# Cancel only pending jobs
scancel --state=PENDING -u $USER

# Cancel by partition
scancel -u $USER --partition=amd
Callout

Be Careful with scancel

Cancellation is immediate and irreversible. Before canceling: 1. Verify the job ID: squeue -j 12345 2. Consider if partial results should be saved 3. Never use scancel -u $USER unless you intend to cancel everything

A safer approach:

BASH 

# Preview what will be cancelled
squeue -u $USER --name=test_job
# Then cancel
scancel --name=test_job

Troubleshooting: Job Stuck in PENDING

Check the reason:

BASH 

squeue -j JOBID -O jobid,reason
Reason Meaning Solution
Resources No suitable node available Reduce resource requests or wait
QOSMaxPerUser Hit quota Wait for other jobs to finish
ReqNodeNotAvail Requested node is down Remove -w flag; let SLURM choose

Troubleshooting: Job Failures

TIMEOUT (exit code 143)

Job exceeded the --time limit.

BASH 

seff JOBID | grep "Job Wall-clock time"
# Resubmit with more time
sbatch --time=08:00:00 script.sh

OUT OF MEMORY

Output contains “Killed” or “Memory limit exceeded.”

BASH 

seff JOBID | grep "Memory"
# Resubmit with more RAM
sbatch --mem=64G script.sh

FAILED (non-zero exit code)

Your code had an error, not a SLURM problem.

BASH 

cat slurm-JOBID.out
cat slurm-JOBID.err
# Fix the error in your code, then resubmit

“Module not found”

Ensure your shebang has the -l flag:

BASH 

#!/bin/bash -l    # MUST have -l

Or explicitly source the module environment:

BASH 

source /etc/profile
module load miniconda3
Challenge

Challenge 1: Cancel Practice

  1. Submit a long-running job:

    BASH 

    sbatch --time=01:00:00 -p amd --wrap="sleep 3600"

  2. Verify it is running: squeue -u $USER

  3. Cancel it: scancel JOBID

  4. Verify cancellation: squeue -u $USER

  5. Check history: sacct -j JOBID --format=jobid,state,exitcode

You should see STATE=CANCELLED.

After canceling, squeue no longer shows the job. sacct shows STATE=CANCELLED with exit code 0:0, because cancellation is not a failure – the job simply did not run to completion.

Key Points
  • Use scancel JOBID to cancel a specific job; scancel -u $USER cancels all your jobs
  • Always verify the job ID before canceling
  • Pending jobs: check reason with squeue -O reason
  • Timeout: increase --time; OOM: increase --mem
  • Script errors show as FAILED with non-zero exit code; check output files
  • Ensure #!/bin/bash -l so modules load correctly in batch scripts

Content from GPU Job Basics

Last updated on 2026-08-24 | Edit this page

Estimated time: 25 minutes

Overview

Questions

  • When should I use a GPU?
  • How do I request GPUs in SLURM?
  • Which GPU type should I choose?
  • How do I write a GPU batch script?

Objectives

  • Understand when GPUs accelerate your work and when they do not
  • Master --gres=gpu syntax for GPU requests
  • Match GPU requirements to available hardware
  • Write and submit GPU batch scripts

When Should You Use a GPU?

Workloads That Benefit

  • Deep Learning: Training neural networks (PyTorch, TensorFlow)
  • Matrix Math: Large matrix multiplications, dense linear algebra
  • GPU-Accelerated Libraries: RAPIDS, CuPy, Dask-CUDA
  • Molecular Dynamics: GPU-accelerated GROMACS, NAMD
  • Custom CUDA Code: Parallel kernels

Workloads That Do NOT Benefit

  • Single-threaded serial code
  • I/O-bound jobs (waiting for disk/network)
  • Light computation (simple statistics, data cleaning)
  • Memory-bound sequential code
Callout

Don’t Waste GPU Resources

Before submitting a GPU job:

  1. Test on CPU first
  2. Run with nvidia-smi and check utilization
  3. If GPU utilization is below 50%, run on CPU instead

GPU nodes are expensive and limited. Wasting them slows everyone else down.

Requesting GPUs

Syntax

BASH 

#SBATCH --gres=gpu:N              # N GPUs, any type
#SBATCH --gres=gpu:TYPE:N         # N GPUs of specific type

Examples

BASH 

#SBATCH --gres=gpu:1              # 1 GPU, any type (fastest to allocate)
#SBATCH --gres=gpu:a100:1         # 1 A100 (80 GB, production)
#SBATCH --gres=gpu:l40s:1         # 1 L40S (48 GB, prototyping)
#SBATCH --gres=gpu:l40s:1         # 1 L40S (48 GB, inference)
#SBATCH --gres=gpu:2              # 2 GPUs, any type

Which GPU to Choose

  • Starting a project: gpu:1 (let SLURM choose)
  • Need lots of GPU memory: gpu:a100:1 (80 GB)
  • Production training: gpu:a100:1 (fastest)
  • Prototyping: gpu:l40s:1 (most available)

GPU Batch Script

BASH 

#!/bin/bash -l
#SBATCH --job-name=train_model
#SBATCH --partition=gpu
#SBATCH --gres=gpu:a100:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=12:00:00
#SBATCH --output=train_%j.out

module load cuda miniconda3
source ~/ml_env/bin/activate

python train.py --epochs 100 --batch_size 128

Key points:

  • --partition=gpu: Must use the gpu partition
  • --gres=gpu:a100:1: Request the GPU
  • --cpus-per-task=8: GPUs still need CPU support threads
  • module load cuda: Required for CUDA libraries

Interactive GPU Testing

Test GPU code before submitting batch jobs:

BASH 

srun --partition=gpu --gres=gpu:l40s:1 \
     --cpus-per-task=4 --mem=16G --time=00:30:00 \
     --pty bash -l

Inside the session:

BASH 

module load cuda miniconda3
python -c "import torch; print(torch.cuda.is_available())"
Challenge

Challenge 1: GPU Test Job

Create and submit a GPU test script:

BASH 

#!/bin/bash -l
#SBATCH --job-name=gpu_test
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=2
#SBATCH --mem=8G
#SBATCH --time=00:05:00
#SBATCH --output=gpu_%j.out

module load cuda miniconda3

python << 'EOF'
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"GPU: {torch.cuda.get_device_name(0)}")
    x = torch.randn(1000, 1000).cuda()
    y = torch.mm(x, x)
    print(f"GPU computation result shape: {y.shape}")
EOF

Submit with sbatch and check the output.

Output should show CUDA available: True, the GPU model name, and a torch.Size([1000, 1000]) result. This confirms your GPU job is configured correctly and the GPU is being used.

Key Points
  • GPUs accelerate parallel computation: deep learning, matrix math, CUDA code
  • Do not use GPUs for serial, I/O-bound, or light computational tasks
  • Request GPUs with --gres=gpu:TYPE:N (e.g., --gres=gpu:a100:1)
  • Always use --partition=gpu and module load cuda for GPU jobs
  • A100 for production, L40S for prototyping and inference, RTX PRO 6000 for large-memory work
  • Test GPU code interactively before submitting long batch jobs

Content from Advanced GPU Workflows

Last updated on 2026-08-24 | Edit this page

Estimated time: 20 minutes

Overview

Questions

  • How do I monitor GPU utilization?
  • How do I write PyTorch and TensorFlow GPU code?
  • How do I run multi-GPU training?
  • How do I troubleshoot GPU errors?

Objectives

  • Monitor GPU utilization with nvidia-smi
  • Write GPU-aware code in PyTorch and TensorFlow
  • Submit multi-GPU training jobs
  • Troubleshoot common GPU errors (CUDA OOM, GPU not detected)

Monitoring GPU Usage

nvidia-smi

From a terminal on the GPU node:

BASH 

nvidia-smi                          # One-time snapshot
watch -n 5 nvidia-smi               # Update every 5 seconds
nvidia-smi dmon                     # Continuous monitoring

Key metrics:

  • GPU-Util: Should be above 80% for well-utilized GPU jobs
  • Memory-Usage: How much of GPU memory is consumed
  • Power: Higher power = busier GPU

GPU Code Examples

PyTorch

PYTHON 

import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Move model and data to GPU
model = torch.nn.Sequential(
    torch.nn.Linear(1000, 500),
    torch.nn.ReLU(),
    torch.nn.Linear(500, 10)
).to(device)

x = torch.randn(1000, 1000).to(device)
output = model(x)

TensorFlow

PYTHON 

import tensorflow as tf

# TensorFlow auto-places on GPU if available
model = tf.keras.Sequential([
    tf.keras.layers.Dense(500, activation='relu', input_shape=(1000,)),
    tf.keras.layers.Dense(10)
])
model.compile(optimizer='adam', loss='mse')
model.fit(x_train, y_train, epochs=10)

Multi-GPU Training

BASH 

#!/bin/bash -l
#SBATCH --job-name=dist_train
#SBATCH --partition=gpu
#SBATCH --gres=gpu:2
#SBATCH --cpus-per-task=16
#SBATCH --mem=128G
#SBATCH --time=12:00:00

module load cuda miniconda3

python -m torch.distributed.launch \
    --nproc_per_node=2 \
    train_distributed.py

Request only multiple GPUs if your code actually supports distributed training.

Troubleshooting GPU Errors

“CUDA out of memory”

Your model or data exceeds GPU memory.

BASH 

# Reduce batch size
python train.py --batch_size 64    # Was 256

# Or request a larger GPU
#SBATCH --gres=gpu:a100:1          # 80 GB instead of the L40S's 48 GB

“GPU not detected” / torch.cuda.is_available() returns False

BASH 

# Verify you requested a GPU
grep gres your_script.sh

# Verify CUDA module is loaded
module list | grep cuda

# Check allocation
squeue -j $SLURM_JOB_ID --format=gres

Low GPU Utilization

Causes: data loading bottleneck, code not on GPU, small batch size.

Fixes:

  1. Add parallel data loading (num_workers in PyTorch DataLoader)
  2. Verify .to(device) or .cuda() in your code
  3. Increase batch size if GPU memory allows
  4. If utilization stays low, use CPU instead

Job Killed for Exceeding Memory

GPU jobs have both CPU RAM and GPU memory limits:

BASH 

# Check which ran out
cat slurm-JOBID.err
seff JOBID | grep Memory
Challenge

Challenge 1: GPU Benchmark

Create a script that benchmarks GPU vs CPU matrix multiplication:

BASH 

#!/bin/bash -l
#SBATCH --job-name=gpu_bench
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=2
#SBATCH --mem=16G
#SBATCH --time=00:10:00
#SBATCH --output=bench_%j.out

module load cuda miniconda3

python << 'EOF'
import torch, time

size = 10000
x_gpu = torch.randn(size, size).cuda()
y_gpu = torch.randn(size, size).cuda()
torch.cuda.synchronize()
start = time.time()
z = torch.mm(x_gpu, y_gpu)
torch.cuda.synchronize()
gpu_time = time.time() - start

x_cpu = x_gpu.cpu(); y_cpu = y_gpu.cpu()
start = time.time()
z_cpu = torch.mm(x_cpu, y_cpu)
cpu_time = time.time() - start

print(f"GPU: {gpu_time:.4f}s, CPU: {cpu_time:.4f}s, Speedup: {cpu_time/gpu_time:.1f}x")
EOF

Submit and compare the GPU vs CPU times.

The GPU should be significantly faster for large matrix multiplication (often 10-100x). The exact speedup depends on the GPU type and matrix size. This demonstrates why GPU acceleration matters for linear algebra and deep learning workloads.

Key Points
  • Monitor GPU utilization with nvidia-smi; aim for over 80% GPU-Util
  • PyTorch uses .to(device) or .cuda(); TensorFlow auto-places on GPU
  • Multi-GPU training requires both --gres=gpu:2 and distributed training code
  • CUDA OOM: reduce batch size or request a larger GPU (A100 has 80 GB)
  • GPU not detected: check --gres, module load cuda, and partition selection
  • Low utilization often means a CPU data-loading bottleneck or undersized batches

Content from Job Arrays and Dependencies

Last updated on 2026-08-24 | Edit this page

Estimated time: 25 minutes

Overview

Questions

  • How can I run many similar jobs efficiently?
  • How do job arrays work?
  • How do I use scratch and tmpfs storage?

Objectives

  • Use job arrays for parameter sweeps and batch file processing
  • Construct array indices and map them to parameters
  • Use scratch and tmpfs storage for fast I/O during jobs

Job Arrays: Parameter Sweeps

A job array submits many similar jobs at once. Each gets a unique task ID via $SLURM_ARRAY_TASK_ID.

Basic Syntax

BASH 

#SBATCH --array=1-10

This submits 10 jobs, numbered 1 through 10.

Example: Learning Rate Sweep

BASH 

#!/bin/bash -l
#SBATCH --job-name=lr_sweep
#SBATCH --array=1-4
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=02:00:00
#SBATCH --partition=amd
#SBATCH --output=sweep_%a.out

module load miniconda3

LRS=(0.001 0.01 0.1 1.0)
LR=${LRS[$((SLURM_ARRAY_TASK_ID - 1))]}

echo "Task $SLURM_ARRAY_TASK_ID: Learning rate = $LR"
python train.py --learning_rate $LR --output results_${LR}.csv

Submit once with sbatch sweep.sh and SLURM creates 4 independent jobs.

File Processing Array

BASH 

#!/bin/bash -l
#SBATCH --array=1-100

FILES=$(ls /bigdata/lab/images/*.jpg | sort)
FILE=$(echo "$FILES" | sed -n "${SLURM_ARRAY_TASK_ID}p")

python process_image.py $FILE

BASH 

#!/bin/bash -l
#SBATCH --array=1-12

# 3 learning rates x 4 batch sizes = 12 combinations
LRS=(0.001 0.01 0.1)
BATCH_SIZES=(32 64 128 256)

LR_IDX=$(( (SLURM_ARRAY_TASK_ID - 1) / 4 ))
BATCH_IDX=$(( (SLURM_ARRAY_TASK_ID - 1) % 4 ))

LR=${LRS[$LR_IDX]}
BATCH=${BATCH_SIZES[$BATCH_IDX]}

python train.py --lr $LR --batch $BATCH

Monitoring Arrays

BASH 

squeue -u $USER              # Shows all array tasks
squeue -j 12345_1            # Check specific task
scancel 12345                # Cancel entire array
scancel 12345_3              # Cancel single task

Fast Storage: Scratch and Tmpfs

Scratch Storage Pattern

Use /scratch for I/O-intensive work (SSD-backed, much faster than network storage):

BASH 

#!/bin/bash -l
#SBATCH --time=02:00:00

WORK_DIR=/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID
mkdir -p $WORK_DIR

# Copy input to scratch
cp /rhome/$USER/large_input.dat $WORK_DIR/

# Work on scratch (fast I/O)
cd $WORK_DIR
python process.py

# Copy results back (scratch is deleted when job completes)
cp output.csv /rhome/$USER/results/

Tmpfs: RAM-Based Storage

For extremely hot data (accessed millions of times):

BASH 

TMPFS_DIR=/tmpfs/$SLURM_JOB_USER/$SLURM_JOB_ID
mkdir -p $TMPFS_DIR
cp hot_data.bin $TMPFS_DIR/
python algorithm.py --input $TMPFS_DIR/hot_data.bin
Callout

Tmpfs Caution

Tmpfs uses RAM. Using too much reduces memory available for your program and can cause the OS to kill processes. Use tmpfs only for small, frequently-accessed data.

Challenge

Challenge 1: Parameter Sweep Array

Create and submit a 5-task job array:

BASH 

#!/bin/bash -l
#SBATCH --job-name=param_sweep
#SBATCH --array=1-5
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --time=00:05:00
#SBATCH --partition=short
#SBATCH --output=array_%a.out

PARAMS=(10 50 100 500 1000)
PARAM=${PARAMS[$((SLURM_ARRAY_TASK_ID - 1))]}

echo "Task $SLURM_ARRAY_TASK_ID: Parameter = $PARAM"
python -c "print(f'Result: {sum(range($PARAM))}')"

Submit with sbatch, then check each output file: cat array_1.out through cat array_5.out.

Five jobs run independently, each with a different parameter value. Results show sum(range(N)) for N = 10, 50, 100, 500, 1000. This demonstrates how arrays parallelize parameter exploration efficiently.

Key Points
  • Job arrays (--array=1-N) submit many similar jobs with one command
  • Each task gets a unique $SLURM_ARRAY_TASK_ID for parameter mapping
  • Use arrays for parameter sweeps, file processing, and grid searches
  • /scratch provides fast SSD storage deleted when the job completes
  • /tmpfs provides RAM-backed storage deleted when the job ends
  • Always copy important results from scratch back to /rhome or /bigdata

Content from Advanced Tips and Troubleshooting

Last updated on 2026-08-24 | Edit this page

Estimated time: 15 minutes

Overview

Questions

  • How can I improve job reliability?
  • How do I check my group’s resource quota?
  • What are common SLURM errors and how do I fix them?
  • How can I optimize job performance?

Objectives

  • Implement checkpoint/resume patterns for long jobs
  • Check group quotas and resource usage
  • Troubleshoot common SLURM errors systematically
  • Apply performance optimization techniques

Improving Job Reliability

Checkpoint/Resume Pattern

For long jobs, save progress so you can restart if interrupted:

BASH 

#!/bin/bash -l
CHECKPOINT=/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID/checkpoint.pkl

if [ -f "$CHECKPOINT" ]; then
    echo "Resuming from checkpoint..."
    python continue_job.py --checkpoint $CHECKPOINT
else
    echo "Starting new job..."
    python start_job.py --save_checkpoint $CHECKPOINT
fi

Output Validation

Verify results before declaring success:

BASH 

python train.py --output model.pkl

if [ ! -f "model.pkl" ]; then
    echo "ERROR: model.pkl not created!"
    exit 1
fi

echo "Validation passed!"

Use set -e

Stop the script on the first error instead of continuing:

BASH 

#!/bin/bash -l
set -e

python step1.py   # If this fails, step2 won't run
python step2.py

Checking Quota and Resource Usage

BASH 

# Check your group's storage and compute quota
quota_check.sh

# Count your running cores
squeue -u $USER --states=RUNNING --format=cpus --noheader | awk '{s+=$1} END {print s}'

# Historical usage (last 7 days)
sacct -u $USER --starttime=$(date -d '7 days ago' +%Y-%m-%d) \
  --format=jobid,jobname,state,elapsed,cpus

Common Errors and Solutions

“Job remains queued indefinitely”

BASH 

squeue -j JOBID -O jobid,reason
  • Resources: No node available – reduce resource requests or wait
  • QOSMaxPerUser: Hit quota – wait for other jobs to finish
  • ReqNodeNotAvail: Specific node is down – remove -w flag

“Permission denied” on Files

BASH 

chmod +x script.sh              # Make script executable
ls -la /bigdata/lab/             # Check permissions
sbatch /rhome/$USER/script.sh   # Use absolute paths

“Module not found”

Ensure shebang is #!/bin/bash -l (the -l flag initializes the module environment).

Performance Optimization

Email Notification Strategy

  • Testing: No email
  • Medium jobs: --mail-type=FAIL
  • Long important jobs: --mail-type=END,FAIL
  • Job arrays: --mail-type=FAIL only (avoid floods)

NUMA Awareness (Large Memory Jobs)

BASH 

#SBATCH --mem-bind=local

CPU Binding (Multi-threaded Code)

BASH 

#SBATCH --cpus-per-task=8
export OMP_PLACES=cores
export OMP_PROC_BIND=close

Monitor Queue Competition

BASH 

squeue --states=PENDING | wc -l
# If > 100, cluster is busy; consider off-peak submission

Complete Production Job Template

BASH 

#!/bin/bash -l
#SBATCH --job-name=production_pipeline
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=12:00:00
#SBATCH --partition=amd
#SBATCH --output=/rhome/$USER/logs/job_%j.out
#SBATCH --mail-user=user@pomona.edu
#SBATCH --mail-type=FAIL,END

set -e

echo "Job started at $(date) on $(hostname), ID: $SLURM_JOB_ID"

module load miniconda3
WORK_DIR=/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID
mkdir -p $WORK_DIR && cd $WORK_DIR

cp /bigdata/lab/input.csv $WORK_DIR/
python /rhome/$USER/src/analyze.py --input input.csv --output results.pkl

cp results.pkl /rhome/$USER/results/results_${SLURM_JOB_ID}.pkl
echo "Job completed at $(date)"
Challenge

Challenge 1: Scratch Storage Benchmark

Compare I/O speed between home and scratch storage:

BASH 

#!/bin/bash -l
#SBATCH --job-name=io_bench
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=00:10:00
#SBATCH --partition=short

WORK_DIR=/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID
mkdir -p $WORK_DIR

dd if=/dev/urandom of=$WORK_DIR/test.bin bs=1M count=100 2>/dev/null

echo "Home storage:"
time dd if=$WORK_DIR/test.bin of=/rhome/$USER/test.tmp bs=1M count=100 2>/dev/null
rm /rhome/$USER/test.tmp

echo "Scratch storage:"
time dd if=$WORK_DIR/test.bin of=$WORK_DIR/test2.bin bs=1M count=100 2>/dev/null

Submit and compare the times.

Scratch should be noticeably faster (often 2-10x) than home storage because it uses local SSDs instead of the network filesystem. This is why scratch is valuable for I/O-heavy jobs.

Key Points
  • Use checkpoint/resume patterns for long jobs that might be interrupted
  • Validate output files before declaring success
  • Use set -e to stop scripts on the first error
  • Check quota with quota_check.sh before large submissions
  • Common fixes: timeout -> more time, OOM -> more RAM, pending -> smaller resources
  • Use scratch for fast I/O; always copy results back to persistent storage
  • Follow the production template: logging, error handling, scratch, validation