Quick Reference: rclone Commands

This reference guide covers essential rclone commands and configuration steps for syncing data between the Sagehen HPC cluster and cloud storage (Box or OneDrive).

Loading rclone on Sagehen HPC

Before using rclone, load the module on Sagehen:

BASH

module load rclone
rclone version

Configuring Remote Storage (OAuth Setup)

Initial Configuration

To configure a new remote (Box or OneDrive), start the interactive setup:

BASH

rclone config

Box Configuration

When prompted, follow these steps:

name> box-storage
Type of storage> box
Box App ID and Secret> (leave blank for Pomona account)
Use Box API> N
Scope that rclone should use> rclone
Advanced config> N
Auto config> Y

A browser window will open for OAuth authentication. Log in with your Pomona credentials and authorize rclone access.

OneDrive Configuration

When prompted:

name> onedrive-storage
Type of storage> onedrive
Client ID and Secret> (leave blank for standard config)
Region> global
Advanced config> N
Auto config> Y

Again, authorize in your browser using your Microsoft account credentials.

View Existing Remotes

BASH

rclone listremotes

Manage Configurations

BASH

rclone config show
rclone config edit

Essential rclone Commands

Listing Files and Directories

List all files in a remote:

BASH

rclone ls box-storage:/

List directories only:

BASH

rclone lsd box-storage:/

List with detailed information:

BASH

rclone ls --human-readable box-storage:/path/to/folder

Copying Files

Copy from remote to Sagehen:

BASH

rclone copy box-storage:/data /rhome/<myusername>/data

Copy from Sagehen to remote:

BASH

rclone copy /rhome/<myusername>/data box-storage:/backup

Copy single file:

BASH

rclone copy box-storage:/results.csv /rhome/<myusername>/

Synchronizing Directories

One-way sync (remote → local):

BASH

rclone sync box-storage:/data /rhome/<myusername>/data

One-way sync (local → remote):

BASH

rclone sync /rhome/<myusername>/data box-storage:/backup

Check what would be synced (dry-run):

BASH

rclone sync --dry-run /rhome/<myusername>/data box-storage:/backup

Mounting Cloud Storage

Mount remote as filesystem:

BASH

rclone mount box-storage:/ ~/box-mount &

List mounted content:

BASH

ls ~/box-mount

Unmount:

BASH

fusermount -u ~/box-mount

Checking File Integrity

Verify copied files match:

BASH

rclone check box-storage:/data /rhome/<myusername>/data

Show differences:

BASH

rclone check --one-way box-storage:/data /rhome/<myusername>/data

Common Flags

Flag	Purpose	Example
--progress	Show transfer progress	rclone copy --progress source dest
--dry-run	Preview changes without executing	rclone sync --dry-run source dest
--transfers N	Parallel transfers (default 4)	rclone copy --transfers 8 source dest
--checkers N	Parallel checkers (default 8)	rclone check --checkers 16 source dest
--verbose	Detailed output	rclone ls --verbose remote:/
--human-readable	Human-friendly file sizes	rclone ls --human-readable remote:/
--exclude PATTERN	Exclude files matching pattern	rclone copy --exclude "*.tmp" source dest
--include PATTERN	Include only matching files	rclone copy --include "*.csv" source dest
--delete-excluded	Delete excluded files from destination	rclone sync --delete-excluded source dest
--no-traverse	Don’t scan destination	rclone copy --no-traverse source dest
--bwlimit RATE	Bandwidth limit	rclone copy --bwlimit 1M source dest

Troubleshooting Token Issues

Token Expiration

If you encounter “invalid token” errors:

BASH

rclone config reconnect box-storage

Then reauthenticate in the browser window that opens.

Re-authorize a Remote

BASH

rclone config edit
# Select your remote and update the token
rclone config reconnect box-storage

Sagehen-Specific Paths

When using rclone on Sagehen, remember storage locations:

• Personal: /rhome/<myusername> (100 GB, backed up)
• Lab/Group: /bigdata/lab/<labname> (1 TB shared)
• Temporary: /scratch/username (no quota, 60-day retention)
• Fast Temp: /tmpfs (RAM-based, job duration only)

Example sync to shared lab storage:

BASH

rclone sync box-storage:/results /bigdata/lab/<labname>/results

Performance Tips

1. Use --transfers for large transfers: rclone copy --transfers 12 source dest
2. Limit bandwidth on shared systems: rclone copy --bwlimit 10M source dest
3. Use /scratch for temporary downloads: Faster SSD storage
4. Monitor transfer status: Add --progress flag
5. Verify large transfers: Use rclone check afterward

Getting Help

View help for any command:

BASH

rclone help command

Full rclone documentation:

BASH

man rclone

HPC Support: - Email: its-hpc@pomona.edu - Instructor: Andrew Wilson