Key Points

Introduction to SLURM

  • A job scheduler coordinates resource sharing on a cluster
  • SLURM is the job scheduler on Sagehen
  • The head node (2 CPUs, 8 GB RAM) is for login and job submission only
  • Jobs run on compute nodes in three partitions: amd (general compute), gpu (GPU-accelerated), and short (quick test/debug jobs)
  • SLURM enforces fair access through queuing, resource limits, and priorities

Partitions and Resources

  • Sagehen has three partitions: amd (general compute), gpu (accelerators), and short (quick / test / debug jobs)
  • Choose partition based on GPU need and job duration
  • GPU partition offers A100 (production), L40S (inference), and RTX PRO 6000 (large-memory training)
  • Per-account resource limits apply; check with quota_check.sh
  • amd and gpu support up to 720 hours (30 days); short has a shorter walltime — check sinfo -p short

Requesting Resources

  • Resource requests use SLURM flags: -c (cores), --mem (memory), -t (time), --gres=gpu (GPUs)
  • Start conservative with estimates; monitor actual usage with seff and adjust
  • Time format: HH:MM:SS or D-HH:MM:SS for longer jobs
  • Use /scratch for fast temporary I/O during jobs; use /rhome or /bigdata for persistent results
  • Over-requesting wastes resources and makes other users wait longer

Interactive Jobs

  • Use srun --pty bash -l for interactive sessions on compute nodes
  • Request resources: -c (cores), --mem (RAM), -t (time), -p (partition)
  • Interactive jobs are best for development, testing, and debugging
  • Use the short partition for short interactive sessions and quick debugging; use amd for longer interactive work
  • Use tmux inside sessions for connection resilience
  • Switch to batch jobs when code is stable and needs long or unattended runs

Writing Batch Scripts

  • Batch scripts are bash scripts with #SBATCH directives for resource requests
  • First line must be #!/bin/bash -l (the -l flag loads your environment)
  • Key directives: --job-name, --cpus-per-task, --mem, --time, --partition, --output
  • Always load modules inside the script, not just interactively
  • Use set -e to stop on first error
  • Include logging (date, hostname, job ID) for debugging
  • Use scratch storage for I/O-intensive intermediate files

Batch Job Submission

  • Submit batch jobs with sbatch script.sh; the returned ID is how you track everything later
  • Status codes: R (running), PD (pending), CG (completing), CD (done), F (failed), TO (timeout)
  • The (REASON) column for pending jobs tells you why they are not running yet
  • Output files use --output and --error directives; %j and %x substitute job ID and name
  • Use tail -f to watch output in real-time while a job runs
  • Email: FAIL for medium jobs, END,FAIL for long ones, nothing for tests, FAIL only for arrays
  • Always run seff JOBID after completion to right-size future submissions
  • You can log out after submission; jobs run unattended

Monitoring Running Jobs

  • Use squeue for real-time job status and watch for continuous monitoring
  • Use sacct to view completed job history and exit codes
  • Use seff to analyze resource efficiency and adjust future requests
  • Exit code 0 = success, 138 = timeout, 143 = cancelled
  • Monitor nodes with free -h, top, and nvidia-smi
  • Use quota_check.sh for BeeGFS storage usage (not du)

Managing and Canceling Jobs

  • Use scancel JOBID to cancel a specific job; scancel -u $USER cancels all your jobs
  • Always verify the job ID before canceling
  • Pending jobs: check reason with squeue -O reason
  • Timeout: increase --time; OOM: increase --mem
  • Script errors show as FAILED with non-zero exit code; check output files
  • Ensure #!/bin/bash -l so modules load correctly in batch scripts

GPU Job Basics

  • GPUs accelerate parallel computation: deep learning, matrix math, CUDA code
  • Do not use GPUs for serial, I/O-bound, or light computational tasks
  • Request GPUs with --gres=gpu:TYPE:N (e.g., --gres=gpu:a100:1)
  • Always use --partition=gpu and module load cuda for GPU jobs
  • A100 for production, L40S for prototyping and inference, RTX PRO 6000 for large-memory work
  • Test GPU code interactively before submitting long batch jobs

Advanced GPU Workflows

  • Monitor GPU utilization with nvidia-smi; aim for over 80% GPU-Util
  • PyTorch uses .to(device) or .cuda(); TensorFlow auto-places on GPU
  • Multi-GPU training requires both --gres=gpu:2 and distributed training code
  • CUDA OOM: reduce batch size or request a larger GPU (A100 has 80 GB)
  • GPU not detected: check --gres, module load cuda, and partition selection
  • Low utilization often means a CPU data-loading bottleneck or undersized batches

Job Arrays and Dependencies

  • Job arrays (--array=1-N) submit many similar jobs with one command
  • Each task gets a unique $SLURM_ARRAY_TASK_ID for parameter mapping
  • Use arrays for parameter sweeps, file processing, and grid searches
  • /scratch provides fast SSD storage deleted when the job completes
  • /tmpfs provides RAM-backed storage deleted when the job ends
  • Always copy important results from scratch back to /rhome or /bigdata

Advanced Tips and Troubleshooting

  • Use checkpoint/resume patterns for long jobs that might be interrupted
  • Validate output files before declaring success
  • Use set -e to stop scripts on the first error
  • Check quota with quota_check.sh before large submissions
  • Common fixes: timeout -> more time, OOM -> more RAM, pending -> smaller resources
  • Use scratch for fast I/O; always copy results back to persistent storage
  • Follow the production template: logging, error handling, scratch, validation