Key Points

Storage Architecture Overview

  • Sagehen provides four storage locations with different persistence and performance characteristics
  • /rhome and /bigdata are persistent and share a 1TB lab quota on BeeGFS
  • /scratch and /tmpfs are temporary, deleted when the job completes
  • Choose storage based on persistence needs, access patterns, and performance requirements

Checking and Managing Quotas

  • Use quota_check.sh for official quota monitoring on Sagehen
  • Always use du --apparent-size on BeeGFS for accurate size measurements
  • Your lab shares a 1TB quota across all members and both /rhome and /bigdata
  • You cannot write files if quota is exceeded – monitor regularly
  • Temporary storage (/scratch, /tmpfs) does not count against quota

Quota Cleanup Strategies

  • Regular cleanup prevents quota-related disruptions
  • Compression can reduce text-based data by 5-20x
  • Use git instead of manual versioning to avoid duplicate copies
  • Always use du --apparent-size to get accurate sizes on BeeGFS
  • Hidden directories like .cache and .local can consume significant space

Transferring Files with OnDemand

  • OnDemand at https://ondemand.hpc.pomona.edu provides GUI-based file management
  • Suitable for small files (under 500 MB) and interactive browsing
  • Always refresh the browser after uploads to verify success
  • Use rsync for large files or batch operations

SFTP Setup with FileZilla

  • FileZilla is a free graphical SFTP client for interactive file transfers
  • Connect to sagehen.hpc.pomona.edu on port 22 using SFTP protocol
  • Duo 2FA is required for all connections
  • Save connection settings in Site Manager for convenience

FileZilla File Operations

  • Drag-and-drop makes FileZilla transfers intuitive for uploads and downloads
  • The transfer queue shows real-time progress, speed, and estimated time
  • Compress files before transfer to reduce time significantly
  • Good for interactive transfers up to several hundred MB; use rsync for larger files

Introduction to rsync

  • rsync is the fastest and most reliable method for large file transfers
  • Always use -P for large files to enable resume capability
  • The trailing slash on source paths is critical: source/ vs source
  • rsync automatically skips unchanged files on subsequent runs
  • Use -avhP as your standard flag combination

Advanced rsync Workflows

  • Use --exclude patterns to skip unwanted files like .git, .pyc, and caches
  • Always use --dry-run before --delete to prevent accidental data loss
  • Compression (-z) helps on slow networks but not for already-compressed files
  • rsync integrates well into SLURM job scripts for staging data to fast storage

Using Scratch Storage in Jobs

  • Scratch (/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID) provides 4-7x faster I/O than /rhome
  • Scratch is deleted when your job completes; always copy results back unconditionally
  • Use the copy-in, compute, copy-out pattern for all I/O-intensive jobs
  • Multi-stage pipelines should keep intermediate files on scratch, not round-trip through NFS
  • Check available space before copying large datasets to fail fast
  • Use a trap with --signal to copy results out even if the job hits its time limit
  • ML training is essentially mandatory on scratch because of DataLoader parallel reads

Using tmpfs RAM Storage

  • /tmpfs is RAM-backed storage providing the fastest I/O on Sagehen HPC
  • Limited by job memory allocation and restricted to single-node jobs
  • Data on /tmpfs is deleted immediately when the job completes
  • Always copy results to permanent storage before the job ends
  • Account for OS overhead when planning tmpfs memory usage

Data Organization Best Practices

  • Organize projects with separate directories for raw data, processed data, code, and results
  • Use meaningful filenames with version numbers or dates
  • Document projects with README files explaining structure and purpose
  • Use /bigdata/lab/<labname> for collaborative data with appropriate permissions
  • Use symbolic links instead of copies for shared reference data

Backup and Archival Strategies

  • Maintain local backups of Sagehen data using rsync scripts
  • Use git for version-controlled backup of code
  • Archive completed projects by compressing and moving to /bigdata
  • Always verify size and file count before deleting the source after migration
  • Plan data retention across the active, archive, and preservation stages

Choosing a Transfer Method

  • Choose transfer method based on file size, frequency, and interface preference
  • OnDemand for small files, FileZilla for interactive work, rsync for everything else
  • rsync is the only method suitable for automation and very large files
  • Compressing data before transfer saves time regardless of method

Troubleshooting and Review

  • Most transfer issues stem from network problems, authentication, or file size limits
  • Most storage issues stem from quota exhaustion or incorrect du usage on BeeGFS
  • Always use quota_check.sh for accurate quota information
  • Temporary storage is deleted when jobs complete – always copy results back
  • Contact for issues you cannot resolve