Summary and Schedule

Cloud Storage Integration with rclone

Welcome to the Cloud Storage Integration with rclone workshop! This course teaches researchers and students at Pomona College how to seamlessly integrate cloud storage (Box and OneDrive) with high-performance computing systems using rclone, a powerful command-line tool for syncing files with cloud providers.

What You’ll Learn

By the end of this workshop, you will be able to:

• Understand the importance of cloud storage integration with HPC systems
• Set up rclone for both Box (institutional storage) and OneDrive (personal storage) at Pomona College
• Execute basic rclone operations: listing, copying, and organizing files
• Implement syncing workflows for moving data between HPC systems and cloud storage
• Automate and maintain cloud storage connections, including token renewal

Who Should Take This Workshop?

This workshop is ideal for:

• Pomona College researchers using the Sagehen HPC cluster
• Students working on computational projects that generate large datasets
• Anyone needing to backup or share research results across systems
• Users who want to automate data transfers between HPC and cloud storage

Prerequisites

You should have:

• A Pomona College NetID and HPC account
• Basic familiarity with the command line (navigating directories, running commands)
• A Pomona College Box account (institutional) or OneDrive account
• SSH access to Sagehen HPC system
• Approximately 4.5 hours for the full workshop

Workshop Structure

This workshop is divided into 12 episodes:

1. Why Cloud Storage Integration Matters: Understand use cases and options
2. Setting Up rclone with Box at Pomona: Step-by-step Box configuration
3. Setting Up rclone with OneDrive at Pomona: Step-by-step OneDrive configuration
4. Basic rclone Operations and Commands: Learn essential rclone syntax and commands
5. Syncing Workflows: Copy vs Sync: Advanced data synchronization strategies
6. Automation and Token Maintenance: Schedule transfers and maintain connections

Key Features of This Course

• Pomona-specific guidance: Uses actual Sagehen HPC infrastructure and institutional accounts
• Hands-on examples: Each episode includes practical challenges with solutions
• Two cloud storage options: Covers both Box (institutional) and OneDrive (personal)
• Real-world workflows: Examples based on actual research scenarios
• Token management: Covers OAuth setup and token renewal procedures

Getting Help

If you encounter issues during this workshop:

• HPC Support: its-hpc@pomona.edu
• Course Contact: Andrew Wilson
• Documentation: See links.md for additional rclone resources

Time Estimates

• Full workshop: ~2.5 hours (teaching + exercises)
• Box setup only: ~1 hour
• OneDrive setup only: ~1 hour
• Individual episodes: 20-45 minutes each

Let’s get started! Begin with Episode 1: Why Cloud Storage Integration Matters.

Acknowledgments

Developed by Andrew Wilson, Director of Research Computing and Digital Scholarship at Pomona College, with Andrei Motchenko, who tested, edited and produced screenshots for the workshop series.

	Setup Instructions	Download files required for the lesson
Duration: 00h 00m	1. Why Cloud Storage Integration Matters	Why would I want to connect cloud storage to an HPC system? What are the benefits of using rclone? What cloud storage options are available at Pomona College?
Duration: 00h 15m	2. Installing and Configuring rclone	How do I load rclone on Sagehen HPC? What is SSH port forwarding and why do I need it? How does the rclone configuration process begin?
Duration: 00h 30m	3. Setting Up Pomona Box	How do I configure rclone for Pomona’s Box storage? How does OAuth authentication work with Box? How do I verify the Box connection?
Duration: 00h 55m	4. Setting Up OneDrive	How do I set up rclone to use OneDrive with my Pomona account? What’s the difference between OneDrive Personal and OneDrive for Business? How do I select the correct drive when multiple options are available?
Duration: 01h 20m	5. Copying Files	How do I copy files from HPC to cloud storage? What does the remote:path syntax mean? How do I preview changes before copying?
Duration: 01h 45m	6. Listing and Inspecting	How do I list files and directories in cloud storage? How can I visualize directory structure? How do I read file contents, delete files, and manage directories?
Duration: 02h 05m	7. Syncing Basics	What is the difference between rclone copy and rclone sync? When should I use sync vs copy? What is a one-way mirror and why is it useful?
Duration: 02h 30m	8. Advanced Sync Workflows	How do I download large datasets from cloud to HPC? How do I implement bidirectional workflows safely? How can I archive deleted files instead of removing them? How do I sync selectively by file type or directory?
Duration: 02h 55m	9. Automating with Cron	How can I schedule automatic syncs? What is cron and how do I create cron jobs? How do I debug scheduled sync jobs?
Duration: 03h 15m	10. SLURM Integration	How do I automatically upload results after a SLURM job finishes? How do I build multi-step pipelines with rclone and SLURM? How do I handle token expiration in automated jobs? What logging do I need for unattended runs?
Duration: 03h 50m	11. Token Management and Maintenance	How do I know when my rclone token has expired? What is token expiration and why does it happen? How do I refresh an expired token?
Duration: 04h 10m	12. Troubleshooting and Best Practices	What should I do if authentication fails? How do I monitor transfer logs and verify successful syncs? What are the best practices for production rclone use?
Duration: 04h 25m	Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.

Pre-Workshop Setup

Before attending the Cloud Storage Integration with rclone workshop, please ensure you have completed the following steps. This will help us make the most of our time together.

Prerequisites

1. Active Pomona College HPC Account

You should have an active account on the Sagehen HPC cluster (sagehen.hpc.pomona.edu). If you haven’t already done so, you can request an account:

• Request form (preferred): HPC account request
• Email: its-hpc@pomona.edu
• Instructor: Andrew Wilson

Your account will provide access to:

• /rhome (100GB personal storage for your files)
• /bigdata (1TB lab storage shared with your research group)
• /scratch (SSD temporary storage for large computations)
• /tmpfs (RAM-based temporary storage for very fast I/O)

Verify your access: Log in to the OnDemand portal at https://ondemand.hpc.pomona.edu/ using your Pomona AD credentials and DUO MFA to confirm your account is active.

2. Cloud Storage Account

You must have at least one of the following accounts:

• Box account (Pomona College provides Box storage for faculty and staff)
• OneDrive account (Microsoft account or institutional)

If you’re unsure whether you have Box access, contact its-hpc@pomona.edu or check https://pomona.box.com.

3. Terminal Familiarity

You should be comfortable with basic terminal commands such as:

• ls (list files and directories)
• cd (change directory)
• pwd (print working directory)
• mkdir (create directories)
• cat (view file contents)

If you need a refresher on command-line basics, we recommend reviewing a basic tutorial before the workshop. Don’t worry; we’ll review essential commands together during the session.

4. Understanding Sagehen HPC Storage

Please familiarize yourself with the different storage locations on Sagehen:

Location	Quota	Purpose	Retention
/rhome/<myusername>	100 GB	Personal, backed up	Permanent
/bigdata/lab/<labname>	1 TB	Lab/group shared	Permanent
/scratch/username	No limit	Temporary compute	60-day deletion policy
/tmpfs	RAM-dependent	Very fast temporary	Job duration only

When syncing with cloud storage, consider which Sagehen location makes sense for your workflow.

What to Have Ready

• Your Pomona username and password
• Your DUO phone for MFA authentication
• Access to your Box or OneDrive account credentials
• A modern web browser (Chrome, Firefox, Safari, or Edge)
• Approximately 500 MB of free disk space on Sagehen for workshop exercises

Technical Setup

Load Required Modules

During the workshop, we’ll use rclone and other command-line tools. You can test your environment by logging in to Sagehen and running:

BASH

module available

This should show available modules on the system. We’ll load the rclone module during the workshop, so don’t worry about this in advance.

Optional: Pre-Install rclone Locally (Advanced)

If you’re interested in using rclone on your personal computer, you can install it before the workshop:

• Windows: Download from https://rclone.org/install/
• macOS: brew install rclone
• Linux: Use your package manager or download from https://rclone.org/install/

This is optional and not required for the workshop.

Questions?

If you encounter any issues with your account, cloud storage access, or need clarification on any prerequisites, please reach out:

• Email: its-hpc@pomona.edu
• Instructor: Andrew Wilson
• Contact: Pomona College HPC Team

We look forward to seeing you in the workshop!