SLURM Job Scheduling on Sagehen HPC

Welcome to Workshop 9: SLURM Job Scheduling on Sagehen

This workshop is designed to teach you how to effectively schedule and manage your computational jobs on Sagehen, Pomona College’s research computing cluster. Whether you’re running machine learning models, statistical analyses, molecular simulations, or other compute-intensive work, understanding SLURM job scheduling is essential to making the most of Sagehen’s powerful computing resources.

What You’ll Learn

By the end of this workshop, you will be able to:

  • Understand the role of job schedulers and why SLURM is used on Sagehen
  • Navigate the Sagehen cluster architecture and available resources
  • Submit interactive jobs for exploratory work and debugging
  • Create and submit batch scripts for long-running computations
  • Monitor and manage your jobs effectively
  • Request and utilize GPU resources appropriately
  • Optimize your job submissions using advanced SLURM features

About Sagehen

Sagehen (sagehen.hpc.pomona.edu) is Pomona College’s research computing cluster managed by Information Technology Services (ITS). The cluster features:

  • Computing nodes: AMD-based compute nodes with 128 CPUs and 500GB RAM each
  • GPU nodes: Specialized nodes with NVIDIA GPUs (A100, L40S, RTX PRO 6000)
  • Storage: Home directories, shared lab storage, and high-speed scratch space
  • Job scheduler: SLURM (Simple Linux Utility for Resource Management)
  • Multi-user environment: Safe resource sharing among hundreds of researchers

Workshop Structure

This 7-episode workshop is organized as follows:

  1. Introduction to SLURM - Learn what job schedulers do and basic SLURM concepts
  2. Partitions and Resources - Explore Sagehen’s available compute partitions and how to choose the right one
  3. Interactive Jobs - Run interactive sessions with srun for immediate feedback
  4. Batch Jobs - Create reproducible batch scripts with sbatch for production runs
  5. Monitoring and Managing - Track and control your submitted jobs
  6. GPU Jobs - Harness the power of Sagehen’s GPU resources
  7. Advanced Tips - Optimize performance with scratch storage, job arrays, and troubleshooting
Prerequisite

Prerequisites

Before starting this workshop, you should have: - Access to Sagehen: An active HPC account (contact if you need one) - SSH client: Ability to connect to a remote system via SSH - Basic Linux knowledge: Comfort with command line operations (files, directories, basic commands) - Text editor: Familiarity with editing text files (nano, vim, or your preferred editor)

If you’re new to Linux or command line work, we recommend completing a Linux basics workshop first.

How to Use This Workshop

  • For learners: Each episode includes hands-on challenges. We recommend working through them on Sagehen as you learn.
  • For instructors: See the Instructor Notes for timing, common issues, and teaching tips.
  • For reference: Check the Quick Reference Card for common SLURM commands.

Getting Help

Questions or issues? Reach out to:

  • Email:
  • Director of Research Computing: Andrew Wilson
  • In-person: Visit ITS in person during office hours

About Carpentries Workbench

This workshop is built using The Carpentries Workbench, a modern lesson development framework that emphasizes hands-on learning and community collaboration.

Learn more about The Carpentries

Ready to get started? Begin with Episode 1: Introduction to SLURM.

Acknowledgments

Developed by Andrew Wilson, Director of Research Computing and Digital Scholarship at Pomona College, with Andrei Motchenko, who tested, edited and produced screenshots for the workshop series.

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.

Getting Started with Sagehen HPC

This page will guide you through setting up access to Sagehen, Pomona College’s research computing cluster. By the end, you’ll be able to connect and start running jobs.

Prerequisites

Before we begin, you need:

  1. Pomona College credentials: Your @pomona.edu email and password
  2. DUO authentication: Access to your Pomona DUO account (for two-factor authentication)
  3. A computer with: Terminal or SSH client (macOS/Linux have this built-in; Windows users can use PowerShell, WSL, or PuTTY)
  4. An HPC account: Contact to request one if you don’t have it

Step 1: Request an HPC Account

If you don’t already have access to Sagehen:

  1. Email with:

    • Your full name
    • Your @pomona.edu email
    • Your department and advisor/PI name
    • Brief description of your research

  2. The HPC team will set up your account (usually 1-2 business days)

  3. You’ll receive confirmation when your account is ready

Step 2: Set Up SSH Access

SSH (Secure Shell) is how you connect to Sagehen. The connection address is:

sagehen.hpc.pomona.edu

On macOS or Linux

  1. Open Terminal

  2. Connect to Sagehen:

    BASH 

    ssh <myusername>@sagehen.hpc.pomona.edu

    Replace username with your Pomona username (part before @pomona.edu)

  3. First time connecting, you’ll see:

    The authenticity of host 'sagehen.hpc.pomona.edu' can't be established.
ECDSA key fingerprint is SHA256:...
Are you sure you want to continue connecting (yes/no/[fingerprint])?

    Type yes and press Enter

On Windows

Option A: Use PowerShell (Windows 10 and newer)

  1. Open PowerShell (search for “PowerShell” in Start menu)

  2. Connect to Sagehen:

    POWERSHELL 

    ssh <myusername>@sagehen.hpc.pomona.edu

Option B: Use WSL (Windows Subsystem for Linux)

  1. Install WSL: Follow Microsoft’s WSL installation guide

  2. Open WSL terminal

  3. Connect to Sagehen:

    BASH 

    ssh <myusername>@sagehen.hpc.pomona.edu

Option C: Use PuTTY

  1. Download PuTTY from https://www.putty.org/
  2. Install and open PuTTY
  3. In the “Host Name” field, enter: sagehen.hpc.pomona.edu
  4. Leave Port as 22
  5. Click “Open”
  6. When prompted, type your username

Step 3: Complete DUO Authentication

When you connect, you’ll see:

Pomona DUO for Sagehen HPC
Duo Authentication

1. Push 'Approve' on your Pomona Duo app
2. Enter an authentication code

Passcode or option (1-3):

Option 1: Use Duo Mobile App (Recommended)

  1. Open the Duo Mobile app on your phone/device
  2. You’ll see a push notification for “Pomona College”
  3. Tap “Approve”
  4. Wait for the app to confirm

Option 2: Enter an Authentication Code

If you don’t have the app or can’t use push:

  1. Type 2 and press Enter to get an SMS code
  2. Check your Pomona-registered phone for a text
  3. Enter the 6-digit code when prompted

Option 3: Backup Code

If you’ve lost access to Duo, you can use a backup code (if you saved one earlier):

  1. Type your backup code
  2. Note: This uses up one of your limited backup codes

Step 4: First Login Test

After authenticating with Duo, you should see:

[<myusername>@sagehen ~]$

Great! You’re connected to the head node.

Verify Your Environment

Run these commands to confirm everything is set up:

BASH 

# Check your username
whoami

# Check which cluster
hostname

# List modules available
module avail

# View available partitions
sinfo

Step 5: Configure Your Environment (Optional)

You can customize your shell environment by editing ~/.bashrc:

BASH 

nano ~/.bashrc

Useful additions:

BASH 

# Set default editor
export EDITOR=nano

# Add helpful aliases
alias sq='squeue -u $USER'
alias sa='sacct -u $USER'
alias ll='ls -lah'

# Set up your favorite modules to auto-load
module load miniconda3  # Load by default

Save with Ctrl+O, Enter, Ctrl+X.

Troubleshooting

“Permission denied (publickey)”

Problem: SSH key issue or wrong credentials

Solution: - Verify username is correct (not email, just the part before @) - Example: if email is , username is alice - Check you’re connecting to correct host: sagehen.hpc.pomona.edu

BASH 

ssh alice@sagehen.hpc.pomona.edu

“DUO authentication failed”

Problem: Two-factor authentication didn’t work

Solutions: - Confirm Duo app is installed and registered with Pomona - Check phone/device for push notification (might be silent) - Try SMS option (type 2 for SMS code) - Verify phone number registered with Pomona is correct - Contact if still having issues

“Connection timeout” or “No route to host”

Problem: Can’t reach sagehen.hpc.pomona.edu

Solutions: - Check internet connection - Verify you can reach other Pomona services (email, Canvas) - If on off-campus network, you may need to use Pomona VPN - Contact to verify your account is active

“Command not found” after login

Problem: Software isn’t installed globally

Solution: Use the module system. See Episode 4 for details.

BASH 

module load miniconda3
python --version

Next Steps

Once connected, you’re ready to begin the workshop:

  1. Start with Episode 1: Introduction to SLURM
  2. Work through each episode in order
  3. Complete the challenges to build your skills
  4. Refer to the Quick Reference Card for command syntax

Getting Help

If you encounter issues:

  1. Technical problems: Email
  2. Questions about this workshop: Ask your instructor
  3. Job failures: Check Episode 7: Monitoring Jobs for troubleshooting

Important Reminders

Callout

Sagehen HPC is a Shared Resource

Remember: - Don’t run jobs on the head node (sagehen.hpc.pomona.edu) - use compute nodes - Close your connection when done: Type exit or logout - Respect quota limits: Check with quota_check.sh - Be a good neighbor: Don’t waste resources on tests, clean up temp files

Your First Session Checklist

Once all these work, proceed to the first episode!