Summary and Setup

Reproducible Research Pipelines

Welcome to the Reproducible Research Pipelines workshop! This is a capstone course designed to equip researchers with the practical skills needed to create, scale, and share computational research workflows that produce consistent, verifiable results.

About This Workshop

Computational research has become increasingly central to scientific discovery, but with that opportunity comes a significant challenge: How do we ensure our results are reproducible? When code runs on different machines, with different software versions, or with subtle variations in how analysis steps are ordered, results can diverge; sometimes subtly, sometimes dramatically.

This workshop teaches you to build research pipelines that:

  • Run the same way every time on your machine, your collaborator’s machine, or an HPC cluster
  • Capture all dependencies so you know exactly what versions of software were used
  • Automate complex workflows so nothing is forgotten or done manually
  • Scale to thousands of jobs on high-performance computing clusters
  • Are easy to understand and modify so your future self (and others) can build on your work
  • Can be shared and published with data and code supporting your publications

Who Is This Workshop For?

This is an advanced workshop designed for researchers who:

  • Have basic Unix shell and Python/R programming skills
  • May have some experience with version control using Git
  • Want to move beyond “it works on my machine” to truly reproducible science
  • Plan to run analyses on HPC clusters like Sagehen

If you haven’t yet used Git or the Unix command line, we recommend starting with The Carpentries Unix Shell Workshop and Git workshop before diving in here.

Workshop Overview

Over the course of this workshop, you will:

  1. Understand reproducibility: What it means, why it matters, and the common obstacles
  2. Organize your research: Implement a standard project structure that separates raw data, code, and results
  3. Capture environments: Use conda and environment files to lock dependencies so your analysis stays reproducible
  4. Automate workflows: Write Makefiles that orchestrate multi-step analyses automatically
  5. Version your research: Use Git not just for code, but for experiments and analysis decisions
  6. Scale to HPC: Harness SLURM job scheduling to run pipelines across compute clusters efficiently
  7. Share and publish: Document your work and release it with DOIs so others can cite and extend your research

Key Technologies

This workshop focuses on tools that are widely available and form the backbone of reproducible research in academia and industry:

  • Git: Version control for code and research decisions
  • conda: Environment management for capturing software dependencies
  • Make: Workflow automation that explicitly encodes analysis steps
  • SLURM: Job scheduling for parallel and high-performance computing
  • Sagehen HPC cluster: The platform we’ll use for practical examples

Learning Outcomes

By the end of this workshop, you will be able to:

  • Explain the three levels of computational reproducibility
  • Design a project structure that supports long-term reproducibility
  • Create conda environments and export them for sharing
  • Write Makefiles that automate research workflows
  • Manage code versions and track experimental branches in Git
  • Submit and orchestrate HPC jobs using SLURM dependencies
  • Create comprehensive documentation for sharing reproducible research
  • Publish reproducible research with proper citations and licensing

How to Use This Material

Each episode contains:

  • Learning objectives at the start
  • Narrative explanations and key concepts
  • Code examples you can run on Sagehen
  • Hands-on exercises for practice
  • Key points summarizing the main ideas

Episodes build on each other, so we recommend working through them in order. However, if you have prior experience with some topics, you can jump to relevant sections.

Prerequisites

Prerequisite

Before starting, you should:

  • Be comfortable with the Unix shell (navigating directories, running commands, redirecting output)
  • Have basic programming experience in Python or R
  • Have access to Sagehen (sagehen.hpc.pomona.edu) or another SLURM-based HPC cluster
  • Have Git installed and a basic understanding of git add, git commit, and git push

If you need a refresher, see the Setup page.

Cluster Setup

All examples assume you’re running on Sagehen with:

  • Lmod module system
  • conda available via module load miniconda3
  • Standard Unix tools (Make, git, etc.)

If you’re using a different cluster, the commands should transfer with minor adjustments to module names or job submission parameters.

Support & Questions

This is an open, community-contributable workshop. If you find errors, have suggestions, or want to add examples relevant to your field, please open an issue or pull request!

License

This material is made available under the Creative Commons Attribution 4.0 license. See LICENSE for details.

Workshop Chapters

The fourteen episodes build one pipeline in stages. Each depends on the one before it, so work through them in order the first time.

Foundations

  1. Why Reproducibility Matters
  2. The Reproducibility Crisis

Organising a project

  1. Project Directory Structure
  2. README and Documentation Files

Capturing your environment

  1. Conda Environment Management
  2. pip, Docker, and HPC Modules

Automating the work

  1. Introduction to Makefiles
  2. Advanced Make Patterns

Tracking changes

  1. Git for Research Projects
  2. Branching and Collaboration

Running it on Sagehen HPC

  1. SLURM Pipeline Basics
  2. Multi-Step SLURM Workflows

Publishing

  1. Sharing Your Research
  2. Putting It All Together

Resources

Acknowledgments

Developed by Andrew Wilson, Director of Research Computing and Digital Scholarship at Pomona College, with Andrei Motchenko, who tested, edited and produced screenshots for the workshop series.

Setup Instructions

Before the Workshop

To get the most out of this workshop, please complete these setup steps before it begins.

Requirements

Local Machine

You need:

HPC Access

  • Access to Sagehen HPC (sagehen.hpc.pomona.edu) or another SLURM-based HPC cluster
  • SSH key pair configured for cluster access
  • Familiarity with ssh and basic command-line navigation

Installation Guide

1. Install git

Mac:

BASH 

# Install Xcode command line tools (includes git)
xcode-select --install

Linux (Ubuntu/Debian):

BASH 

sudo apt-get install git

Windows:

Verify:

BASH 

git --version
# Should show: git version X.X.X

2. Install conda

Download Miniconda from: https://docs.conda.io/projects/miniconda/en/latest/

Mac (Intel):

BASH 

curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-MacOSX-x86_64.sh

Mac (Apple Silicon/M1/M2):

BASH 

curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
bash Miniconda3-latest-MacOSX-arm64.sh

Linux:

BASH 

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

Windows:

Verify:

BASH 

conda --version
# Should show: conda X.X.X

3. Install Make

Mac:

BASH 

# Included with Xcode command line tools
xcode-select --install

Linux (Ubuntu/Debian):

BASH 

sudo apt-get install build-essential

Windows (WSL2):

BASH 

# Within WSL2 Ubuntu environment
sudo apt-get install build-essential

Verify:

BASH 

make --version
# Should show: GNU Make X.X.X

4. Set Up HPC Access

Test your connection to Sagehen:

BASH 

ssh <myusername>@sagehen.hpc.pomona.edu

You should see a welcome message and command prompt. If you get permission denied:

  1. Verify your username is correct
  2. Check that your SSH key is set up (contact HPC support if needed)
  3. Check your firewall/VPN if off-campus

From Sagehen, verify the module system works:

BASH 

module list
module load miniconda3
conda --version

Creating Your First Environment

Test that you can create and use a conda environment:

BASH 

# Create an environment
conda create -n test-env python=3.11 numpy pandas

# Activate it
conda activate test-env

# Verify it works
python -c "import numpy, pandas; print('Success!')"

# Deactivate
conda deactivate

Verify Your Setup

Run this script to check everything is working:

BASH 

#!/bin/bash

echo "Checking prerequisites..."

# Check git
if command -v git &> /dev/null; then
    echo "✓ git: $(git --version)"
else
    echo "✗ git not found"
fi

# Check conda
if command -v conda &> /dev/null; then
    echo "✓ conda: $(conda --version)"
else
    echo "✗ conda not found"
fi

# Check make
if command -v make &> /dev/null; then
    echo "✓ make: $(make --version | head -1)"
else
    echo "✗ make not found"
fi

# Check HPC access
if ping -c 1 sagehen.hpc.pomona.edu &> /dev/null; then
    echo "✓ Network access to sagehen.hpc.pomona.edu"
else
    echo "✗ Cannot reach sagehen.hpc.pomona.edu (may be expected if off-campus)"
fi

echo ""
echo "All prerequisites ready!"

Save as check_setup.sh, run with bash check_setup.sh

Troubleshooting

conda: command not found

Your shell doesn’t know about conda. After installing, you need to restart your terminal or run:

BASH 

# Reload your shell configuration
source ~/.bashrc
# or on Mac:
source ~/.zprofile

git: command not found (Mac)

Install Xcode command line tools:

BASH 

xcode-select --install

Permission denied (publickey) for sagehen

Contact HPC support () to verify:

  • Your account is active
  • Your SSH key is installed
  • You have access to Sagehen

Command not found: make

Install build-essential (Linux) or Xcode tools (Mac):

Linux:

BASH 

sudo apt-get update
sudo apt-get install build-essential

Mac:

BASH 

xcode-select --install

Getting Help

Next Steps

Once you’ve completed setup:

  1. Clone the workshop materials (instructions will be provided)
  2. Review the Learner Profiles to see if you fit a described profile
  3. Skim the Reference for an overview of key concepts
  4. You’re ready to begin Episode 1!