Key Points

Why Cloud Storage Integration Matters

  • Cloud storage integration solves real HPC data transfer challenges (sharing, backup, off-campus access)
  • rclone is a powerful command-line tool for syncing files between HPC and cloud storage
  • Pomona provides two cloud options: Box (institutional) and OneDrive (personal)
  • OAuth authentication provides secure token-based access without storing passwords
  • Always consider data governance and security policies before moving data to cloud storage

Installing and Configuring rclone

  • SSH port forwarding (-L 53682:localhost:53682) enables secure OAuth authentication between your browser and Sagehen
  • The rclone module must be loaded with module load rclone before use on Sagehen
  • rclone configuration is stored securely in ~/.config/rclone/rclone.conf and should never be shared
  • The rclone config command starts an interactive setup wizard for adding cloud storage remotes

Setting Up Pomona Box

  • Box uses Pomona’s Single Sign-On (SSO) for authentication, which may include two-factor authentication
  • rclone configuration stores OAuth tokens securely in ~/.config/rclone/rclone.conf and does not store your password
  • Test your configuration with rclone about and rclone lsd commands to verify the connection works
  • The rclone remote name (e.g., pomona-box) is your local reference for this Box account and can be customized

Setting Up OneDrive

  • Pomona’s institutional OneDrive for Business is the supported endpoint.
  • The Microsoft OAuth flow requires signing in with your Pomona email and handling two-factor authentication
  • You can configure multiple cloud remotes with rclone (Box and OneDrive) and reference them with their unique names
  • Test your OneDrive configuration with rclone about and rclone lsd commands just like Box

Copying Files

  • The remote:path syntax (e.g., pomona-box:/folder/file.txt) is fundamental to all rclone commands
  • rclone copy is additive and safe – it never deletes files at the destination
  • The –dry-run flag previews changes without making them and should always be used before large operations
  • Understanding the difference between local paths (~/file.txt) and remote paths (remote:path) is critical for avoiding mistakes

Listing and Inspecting

  • rclone ls lists files, rclone lsd lists directories, and rclone tree shows the complete folder structure
  • rclone cat displays file contents without downloading, useful for previewing data
  • rclone delete removes files (not folders) while rclone purge removes folders and all contents
  • Always preview destructive operations with –dry-run before executing them

Syncing Basics

  • rclone copy is safe for backups (additive only), while rclone sync creates an exact mirror and deletes files not in the source
  • Always preview sync operations with –dry-run before executing to avoid accidental data loss
  • Use rclone copy for backups and rclone sync only for one-way mirroring where you want the destination to exactly match the source
  • Reversing source and destination in rclone sync is a common and dangerous mistake

Advanced Sync Workflows

  • One-way mirroring (sync from HPC to cloud) is appropriate for daily results; use separate folders for bidirectional workflows
  • The –backup-dir flag preserves deleted files in an archive instead of permanently removing them during sync
  • Selective syncing with –include and –exclude filters allows you to sync only specific file types or directories
  • Bandwidth limiting with –bwlimit prevents large transfers from overwhelming the shared network

Automating with Cron

  • Cron jobs require absolute file paths and may need module load rclone in the command
  • Shell scripts with error handling and email notifications make automated syncs more robust
  • Always test cron jobs by running the command manually first before scheduling
  • Check cron logs regularly to catch token expiration and other failures early

SLURM Integration

  • SLURM job scripts can include rclone copy/sync commands to automatically upload results after a job finishes
  • Multi-step pipelines can download data from cloud, process it on HPC, and upload results back to cloud
  • Always verify token validity before automated uploads to avoid silent failures
  • Use –log-file to capture transfer details for debugging failed uploads
  • Pack many small files into a tarball before upload to avoid per-file overhead
  • Use rclone sync only when you intend destination-pruning; rclone copy is safer
  • Always include output verification before uploading to catch silent pipeline failures

Token Management and Maintenance

  • OAuth tokens expire every 2-3 months and should be refreshed proactively with rclone config reconnect
  • Use rclone about remote: to test token validity before running important automated jobs
  • Token renewal requires SSH port forwarding (-L 53682:localhost:53682) just like initial setup
  • Set calendar or cron reminders to refresh tokens before they expire

Troubleshooting and Best Practices

  • Common errors like “connection refused” and “command not found” have straightforward fixes related to port forwarding and module loading
  • Monitor rclone logs with –log-file to debug failed transfers and track successful synchronizations over time
  • Use rclone size and rclone about to check data volumes and available space before large transfers
  • Contact for HPC-specific issues and consult rclone.org/docs for rclone documentation