Summary and Schedule

Data Transfer and Storage Management on Sagehen

Welcome to Workshop 12: Data Transfer and Storage Management for Pomona College’s Sagehen HPC cluster.

Overview

In this workshop, you will learn how to:

  • Navigate Sagehen’s hierarchical storage system
  • Understand storage quotas and monitor your disk usage
  • Transfer files efficiently using multiple methods
  • Use temporary storage for high-performance computing jobs
  • Choose the right tool for different transfer scenarios
  • Implement best practices for data organization and backup

Target Audience

This workshop is designed for researchers and students who need to:

  • Work with data on the Sagehen HPC cluster
  • Transfer files to and from Sagehen
  • Manage their storage allocation effectively
  • Optimize file transfer performance

Learning Outcomes

After completing this workshop, you will be able to:

  1. Describe the Sagehen storage hierarchy and when to use each filesystem
  2. Check your storage quotas and manage disk space
  3. Transfer files using the web interface, SFTP, and rsync
  4. Copy data to/from temporary storage in job scripts
  5. Select appropriate transfer methods for different scenarios
  6. Implement data organization and backup strategies

Prerequisites

  • Active Sagehen account with OnDemand access
  • Basic familiarity with Linux/Unix command line (helpful for rsync episodes)
  • SSH client software (built-in on macOS/Linux, use Windows Subsystem for Linux on Windows)
  • FileZilla installed (optional, for SFTP episodes)

Cluster Information

  • Cluster Name: Sagehen
  • File System: BeeGFS (parallel distributed filesystem)
  • Access: OnDemand web interface or SSH
  • Support Email:
  • Support Contact: Andrew Wilson

Quick Reference

Storage Locations

Path Purpose Quota Persistence
/rhome/<myusername> Home directory Lab quota Permanent
/bigdata/lab/<labname> Shared lab storage 1TB per lab Permanent
/scratch/$SLURM_JOB_ID Job-local SSD temp Job time limit Deleted after job
/tmpfs/$SLURM_JOB_ID RAM-backed temp Job memory Deleted after job

Transfer Methods

Method Use Case Speed Graphical
OnDemand Web UI Small files (<100 MB) Slow Yes
FileZilla SFTP Interactive transfer Medium Yes
rsync Large files, sync Fast No
rclone Cloud storage Medium Via OnDemand

Workshop Structure

This workshop consists of 14 episodes covering progressively more advanced topics:

  1. Storage Hierarchy - Understand the filesystem layout
  2. Quotas Management - Monitor and manage your storage
  3. OnDemand Transfers - Use the web interface
  4. SFTP with FileZilla - Graphical file transfers
  5. rsync CLI - Command-line bulk transfers
  6. Scratch and Tmpfs - Temporary storage in jobs
  7. Best Practices - Storage strategy and organization

Estimated total time: 4-5 hours

Getting Help

  • Before Workshop: Review the setup instructions in the Setup page
  • During Workshop: Instructors will guide you through each episode
  • After Workshop: Refer to the Reference page for command summaries
  • Technical Support: Contact

Last Updated: March 5, 2026 License: CC-BY 4.0 Carpentry: The Carpentries Incubator

Acknowledgments

Developed by Andrew Wilson, Director of Research Computing and Digital Scholarship at Pomona College, with Andrei Motchenko, who tested, edited and produced screenshots for the workshop series.

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.

Setup Instructions for Data Transfer and Storage Management Workshop

Before the Workshop

To get the most out of this workshop, please complete these setup steps before attending.

Prerequisites

This workshop assumes you have:

  • An active Sagehen HPC cluster account
  • Basic familiarity with Linux/Unix command-line interface
  • A personal computer (Windows, macOS, or Linux)
  • Internet connection

If you don’t have a Sagehen HPC account, request one at least one week before the workshop via the HPC account request form (or by emailing )

Software Installation

1. SSH Client

macOS/Linux: SSH is pre-installed. Test by opening Terminal and running:

BASH 

ssh -V
# Should show OpenSSH version

Windows: - Option A: Use Windows Subsystem for Linux (WSL 2) - Install from Microsoft Store - Or follow: https://docs.microsoft.com/en-us/windows/wsl/install

  • Option B: Use Windows 10/11 built-in OpenSSH
    • Settings → Apps → Optional Features
    • Search “OpenSSH” and install
    • Test in PowerShell: ssh -V
  • Option C: Install PuTTY

Used in Episode 4 for graphical file transfers.

Download: https://filezilla-project.org/download.php?type=client

Installation: - Windows: Run .exe installer - macOS: Open .dmg and drag to Applications - Linux: Use package manager (apt, dnf, etc.)

3. rsync

Used in Episode 5 for bulk file transfers.

macOS: Pre-installed. Test with:

BASH 

rsync --version

Linux: Pre-installed on most distributions. Test with:

BASH 

rsync --version

Windows: - Install via WSL (recommended): sudo apt-get install rsync - Or use Cygwin with rsync package

4. Web Browser

For OnDemand web interface in Episode 3. - Chrome, Firefox, Safari, or Edge (all supported) - Ensure JavaScript is enabled

Account and Access Verification

1. Verify Sagehen HPC Account Access

Test SSH connection to Sagehen:

BASH 

ssh <myusername>@sagehen.hpc.pomona.edu
# Enter password when prompted
# Complete Duo 2FA authentication
# If successful, you'll see Sagehen login prompt

Type exit to log out.

2. Verify OnDemand Access

Test OnDemand web interface:

  1. Open browser
  2. Navigate to: https://ondemand.sagehen.hpc.pomona.edu
  3. Login with Pomona credentials
  4. Complete Duo authentication
  5. You should see OnDemand dashboard

3. Verify Quota

Check your storage allocation:

BASH 

ssh <myusername>@sagehen.hpc.pomona.edu
quota_check.sh
# Should show your current usage and quota
exit

Create Test Data

Create some test files for the workshops:

BASH 

# Create test directory
mkdir ~/test_data
cd ~/test_data

# Create small test file (for FileZilla and OnDemand examples)
echo "This is a test file" > small_file.txt

# Create medium test file (for rsync examples)
dd if=/dev/zero of=medium_file.bin bs=1M count=10
# Creates 10 MB file

# Create sample CSV data
cat > sample_data.csv << 'EOF'
id,name,value
1,sample_a,100
2,sample_b,200
3,sample_c,300
EOF

Option 2: Use Existing Files

If you prefer, use any existing files on your computer: - Documents (PDF, text) - Data files (CSV, JSON) - Images (JPG, PNG)

Test File Transfer

Before the workshop, test one file transfer method:

BASH 

cd ~/test_data

# Upload test file
rsync -avhP small_file.txt <myusername>@sagehen.hpc.pomona.edu:/rhome/<myusername>/

# You should see:
# Connecting via SSH...
# Duo prompt (approve on phone)
# Transfer progress
# File listed on remote

# Verify on Sagehen
ssh <myusername>@sagehen.hpc.pomona.edu ls -lh /rhome/<myusername>/small_file.txt

Test FileZilla (If Installed)

  1. Open FileZilla
  2. Enter in Quick Connect:
    • Host: sagehen.hpc.pomona.edu
    • Username: your_username
    • Port: 22
  3. Click Quickconnect
  4. Approve Duo
  5. Should see home directory in right panel

Test OnDemand

  1. Navigate to: https://ondemand.sagehen.hpc.pomona.edu
  2. Click “Files”
  3. Should see your home directory

Before the workshop, create a project directory for workshop exercises:

BASH 

mkdir -p ~/workshop_data/incoming ~/workshop_data/results
# incoming: For files we'll transfer INTO Sagehen
# results: Where we'll download results FROM Sagehen

Troubleshooting Setup

Cannot Connect via SSH

Problem: “Connection refused” or “Network unreachable”

Solutions: 1. Verify Sagehen is not down (check email) 2. Check hostname: sagehen.hpc.pomona.edu 3. Check internet connection: ping google.com 4. If behind corporate firewall, port 22 may be blocked 5. Contact for help

Duo Authentication Not Working

Problem: No Duo prompt appears or authentication times out

Solutions: 1. Check phone is nearby and powered on 2. Check Duo app is installed and updated 3. Try again (may be delayed) 4. Check your Duo enrollment: https://accounts.pomona.edu/ 5. Contact IT:

FileZilla Won’t Install

Problem: Permission error or installation fails

Solutions: 1. Run installer as administrator (Windows) 2. For macOS, drag to Applications folder in a new Finder window 3. Use package manager instead (Linux) 4. FileZilla is optional; OnDemand and command-line work without it

rsync Not Found

Problem: “command not found” when typing rsync

Solutions: 1. macOS: Pre-installed, try full path: /usr/bin/rsync 2. Linux: Install with sudo apt-get install rsync or equivalent 3. Windows: Install WSL first, then rsync via WSL 4. rsync is optional; FileZilla works without it

Bandwidth and Network Notes

This workshop involves file transfers. If on a slow connection:

  • Slow connection? Download large test file overnight or before workshop
  • Limited bandwidth? Skip large file transfer examples (instructor can demo)
  • On cellular? Use WiFi for large transfers

What to Bring

  • Laptop with SSH client (Windows, macOS, or Linux)
  • Pomona account credentials
  • Phone for Duo authentication
  • Notes/notebook for recording commands
  • Optional: FileZilla if you want to follow graphical examples

Getting Help

Before Workshop

If you encounter setup issues:

  1. Check email: Setup instructions may have updates
  2. Contact HPC Support:
  3. Contact IT: (for account/Duo issues)

During Workshop

Instructors will help with any setup issues at the start of the workshop. Allow 15 minutes for everyone to verify access.

Optional Pre-Workshop Review

To prepare for the workshop, consider reviewing:

  1. Basic Linux commands: https://ubuntu.com/tutorials/command-line-for-beginners
  2. SSH explained: https://www.ssh.com/ssh/command
  3. File transfer concepts: https://en.wikipedia.org/wiki/Secure_file_transfer_protocol

Ready?

If you’ve completed these steps, you’re ready for the workshop! See you there.

Questions? Email