Summary and Schedule

Discover how to apply machine learning to complex research problems using the computational power of Sagehen HPC. This workshop introduces practical machine learning on HPC systems, covering environment setup, data preparation, training models with PyTorch and TensorFlow, and managing long-running training jobs. You’ll learn to scale your machine learning workflows to leverage multiple CPU cores and GPU acceleration. This workshop is part of the Pomona College HPC Workshop Series.

Prerequisite

Prerequisites

This workshop is part of the Pomona College Research Computing Workshop Series for researchers using the Sagehen HPC cluster.

Before starting, ensure you have: - Completed Workshop 16 (GPU Computing) or equivalent GPU experience - Python proficiency required - A Sagehen HPC account (request from its-hpc@pomona.edu) - Access to the OnDemand portal

For questions or support, contact the Research Computing team at its-hpc@pomona.edu.

Designed for researchers new to machine learning on HPC systems, this workshop assumes basic Python knowledge and familiarity with HPC concepts. You will gain practical skills in building, training, and optimizing machine learning models on Sagehen’s computational resources.

Acknowledgments

---

Developed by Andrew Wilson, Director of Research Computing and Digital Scholarship at Pomona College, with Andrei Motchenko, who tested, edited and produced screenshots for the workshop series.

	Setup Instructions	Download files required for the lesson
Duration: 00h 00m	1. Why Machine Learning on HPC?	When do you need HPC for machine learning? What problems require cluster computing? What are the advantages and tradeoffs of HPC for ML? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 00h 15m	2. ML Workflow Overview	How do CPUs and GPUs compare for different ML workloads? What GPU options are available on Sagehen HPC? What does a typical ML workflow on HPC look like? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 00h 30m	3. Setting Up Your ML Environment	How do you load ML modules on Sagehen HPC? What pre-built modules are available for PyTorch and TensorFlow? How do you verify GPU access is working? When should you reuse a shared conda environment vs. build your own? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 01h 05m	4. Loading ML Frameworks	How do you create custom conda environments for ML? How do you create reproducible environments for team collaboration? What are common environment setup mistakes and how do you fix them? How do you keep environments small enough to fit your /rhome quota? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 01h 40m	5. Data Preparation Basics	How do you efficiently prepare large datasets on HPC systems? What are best practices for handling missing values, normalization, and encoding? How do you avoid common data preparation pitfalls? When does pandas hit its limits, and what comes next? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 02h 25m	6. Data Pipelines for HPC	How do you process datasets larger than RAM? How do you build efficient data loading pipelines for GPU training? How do you run preprocessing at scale with SLURM? When should preprocessing run on CPU nodes vs GPU nodes? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 03h 10m	7. Training with PyTorch	How do you set up PyTorch for GPU training on Sagehen HPC? What does an efficient training loop look like? How do you handle GPU memory errors? How do you choose between A100, L40S, and RTX PRO 6000 nodes for your job? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 04h 05m	8. Training with TensorFlow	How do you set up TensorFlow for GPU training on Sagehen HPC? What are the differences between PyTorch and TensorFlow workflows? How do you use Keras for rapid model development? When should you choose TensorFlow over PyTorch (and vice versa)? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 04h 50m	9. Checkpointing Long Jobs	How do you save and resume long-running training jobs? What should a complete checkpoint include? How do you implement multi-GPU training on Sagehen HPC? How often should you checkpoint, and where should checkpoints live? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 05h 35m	10. SLURM Scripts for ML	How do you write SLURM scripts for GPU training jobs? How do you request GPU resources correctly? How do you launch multi-GPU training with SLURM? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 05h 55m	11. Hyperparameter Tuning	How do you efficiently tune hyperparameters on HPC? What are the differences between grid search, random search, and Bayesian optimization? How do you parallelize hyperparameter sweeps with SLURM? How do you avoid wasting GPU-hours on bad runs? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 06h 40m	12. ML Best Practices	How do you ensure reproducibility of ML experiments? What are common pitfalls in HPC ML workflows? How do you track experiments and deploy models? What habits separate research-ready ML from one-off scripts? ::::::::::::::::::::::::::::::::::::::::::::::::
Duration: 07h 20m	Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.

Setup: Introduction to Machine Learning on HPC

Before the Workshop

Computer and Internet

• A laptop or desktop with internet connection
• SSH access to Sagehen HPC (Pomona’s HPC cluster)
• A terminal application (Terminal on Mac, PowerShell or WSL on Windows)

Accounts

• Active Pomona College account
• Access to Sagehen HPC system
  • Request at: its-hpc@pomona.edu
  • Takes 1-2 business days to set up

Optional: Local Environment

For quick testing before running on GPU: - Install Python 3.10+ - Install pip - Optional: Miniconda for conda environments

During the Workshop

Access Sagehen HPC

BASH

# Log in
ssh your_netid@sagehen.hpc.pomona.edu

# There are no pytorch/tensorflow modules on Sagehen -- ML frameworks
# live in conda environments (created in episode 4):
module load miniconda3
conda activate pytorch_env    # or tf_env

Download Workshop Materials

BASH

# The example data and scripts live in the lesson repository
git clone https://github.com/Pomona-College/hpc-intro-ml.git
cd hpc-intro-ml/episodes/data

You only need this if you want the example datasets locally. Everything in the episodes can also be typed directly, and the small sample files are reproduced inline where they are used.

Create Your Working Directory

BASH

# Create personal workspace in your home directory
mkdir -p ~/ml-workshop
cd ~/ml-workshop

# Create subdirectories
mkdir -p data models logs

/scratch is not writable from the login node — mkdir -p /scratch/$USER/... there fails with Permission denied. Create scratch directories inside a job (srun or sbatch) only, and treat anything left there as gone once the job ends.

Hardware Available on Sagehen HPC

Sagehen provides 10 GPUs across A100/L40S/RTX PRO 6000 generations (confirmed May 2026); see Workshop 16 for full hardware details.

GPU Nodes

• A100 (80 GB, ×4): most powerful, for the largest models
• L40S (48 GB, ×4): good for standard deep learning
• RTX PRO 6000 (96 GB ECC, ×2): the largest GPU memory on Sagehen; very large models and pro workloads

Storage

• /rhome: 100 GB (your home, slower, backed up)
• /bigdata: Large group storage (moderate speed)
• /scratch: Fast node-local SSD, writable only inside a job and deleted when the job ends

Example Specs

• Per GPU node: 4 CPUs, 32 GB system RAM
• Partition: gpu (for GPU jobs)
• Typical limits: 4 GPUs per account, 2-hour default job limit

Software Stack

Pre-installed Modules

BASH

# Check available modules
module avail

# Modules you will actually use:
module load miniconda3          # Python/conda (frameworks install into conda envs)
module load anaconda3           # alternative conda distribution
module load cuda/12.2.1         # CUDA toolkit (current default)

Python Packages

Common ML packages available in conda: - PyTorch, TensorFlow, scikit-learn - NumPy, pandas, matplotlib - Jupyter notebooks (with GPU support via OnDemand)

Getting Help

During Workshop

• Instructors and TAs available
• Check #ml-hpc channel on community Slack (if available)

After Workshop

• HPC Support: its-hpc@pomona.edu
• Sagehen documentation: the Pomona College HPC Workshop Series — start with Introduction to HPC Systems
• Pomona ITS: https://www.pomona.edu/its/
• Framework Documentation:
  • PyTorch: https://pytorch.org
  • TensorFlow: https://tensorflow.org
  • Scikit-learn: https://scikit-learn.org

Troubleshooting Common Issues

Can’t connect to Sagehen? - Check VPN is connected - Verify NetID and password - Contact ITS at its-hpc@pomona.edu

GPU not available? - Check: squeue -u $USER - May need to wait for job queue - Start with smaller job to test

Module not found? - Run: module avail - Try: module load cuda/11.8.0 (a different version) - Ask instructors

Workshop Outline

1. Episode 1: Why Machine Learning on HPC?
2. Episode 2: Environment Setup (conda, modules)
3. Episode 3: Data Preparation and Preprocessing
4. Episode 4: GPU Training with PyTorch
5. Episode 5: GPU Training with TensorFlow/Keras
6. Episode 6: Distributed Training Across GPUs
7. Episode 7: Hyperparameter Tuning and Best Practices

Recommended Reading

• Sagehen Quick Start Guide (available in first episode)
• PyTorch GPU Tutorial (link in Episode 4)
• TensorFlow Performance Guide (link in Episode 5)