Key Points

Why Encrypt? Regulations and Risk

  • Encryption is required for FERPA, HIPAA, EAR/ITAR, and NIST CUI-protected data
  • A single breach cascades: notification → investigation → fines → reputation damage → career impact
  • Higher education is a persistent breach target; 500+ million records exposed in 5 years
  • FERPA violations cost $60,000+ per violation; HIPAA violations $100–$50,000 per record
  • Pomona ITS Policy 24 mandates encryption for RESTRICTED data
  • Encryption stops preventable breaches like storage misconfigurations
  • Back up gocryptfs.conf separately; it cannot be recovered without the file

Pomona's Data Classification System

  • Pomona’s three-tier system: PUBLIC (no encryption), PROPRIETARY (recommended), RESTRICTED (mandatory)
  • RESTRICTED data is legally protected under FERPA, HIPAA, EAR, and NIST requirements
  • gocryptfs protects data at rest and prevents accidental exposure from misconfiguration
  • Encryption is one layer; defense-in-depth requires strong passphrases, SSH keys, and MFA
  • Encryption supports research integrity, publication, and funding compliance
  • If you have doubt about data classification, encrypt it

Scenarios and Responsibilities

  • Real-world scenarios: IACUC, FERPA, EAR, and NDA data all require encryption
  • You classify and manage data; ITS provides tools; Compliance enforces policy
  • Breach response: Stop, preserve evidence, contact ITS, inform advisor
  • Data classification rule: Identifiers + sensitivity = RESTRICTED = encrypt
  • Passphrase access: Only you (and advisor for emergency backup)
  • Report suspected breaches immediately; early reporting is safer than delayed discovery

gocryptfs Architecture

  • gocryptfs is the secure successor to EncFS, created in 2014 after security vulnerabilities in EncFS
  • The two-directory model: cipher (encrypted on-disk) and plain (decrypted in-memory during mount)
  • FUSE (Filesystem in Userspace) allows unprivileged users to create and manage filesystems
  • Encryption is transparent: applications use regular file operations; gocryptfs handles encryption invisibly
  • File-level encryption (not block-level) is necessary for HPC shared storage (no admin required)
  • The cipher directory is always encrypted; the plain directory exists only during an active mount session
  • FUSE makes encryption invisible to applications—they use open(), read(), write() normally

Encryption, Performance, and Comparison

  • AES-256-GCM is unbreakable: 2^256 possible keys, authenticated encryption, hardware-accelerated
  • scrypt key derivation is memory-hard: slows down password cracking by making it time-consuming per guess
  • gocryptfs.conf is irreplaceable: losing it means losing all data, even with passphrase
  • File-level encryption (not block-level) is necessary for HPC shared storage (no admin required)
  • Filename encryption prevents information leakage even to users with filesystem access
  • Performance impact is negligible: ~5-10% overhead, hidden by computation in typical research workloads
  • gocryptfs is optimal for HPC clusters: secure, accessible, practical, maintained, cluster-friendly
  • gocryptfs DOES protect against: disk theft, misconfiguration, backup breach, casual access
  • gocryptfs DOES NOT protect against: active malware, SSH compromise with mount active, memory attacks, root access
  • Defense-in-depth: encryption + strong passphrases + SSH keys + MFA + safe practices

Planning Your Encrypted Directories

  • Plan directory structure first: use naming conventions to avoid confusion
  • Choose storage location based on dataset size: /bigdata for large (>100 MB), /rhome for small (<100 MB)
  • Create strong passwords: 14+ characters, mixed types, memorable or in password manager
  • Always use quota_check.sh on BeeGFS—du gives wrong results
  • Encryption overhead is minimal (~1% per file)
  • Never store encrypted directories in /scratch, /tmpfs, or root directories
  • Create both cipher and plain directories before initialization
  • Password cannot be recovered—choose carefully and store securely

Verifying and Managing Encrypted Directories

  • Initialization creates gocryptfs.conf (encrypted key) and gocryptfs.diriv (directory IV)
  • Verify encryption: encrypted files unreadable in cipher directory, readable in plain directory
  • Directory permissions must be 700 (drwx——) on both cipher and plain directories
  • Back up gocryptfs.conf immediately in multiple locations—it’s your only access key
  • Common mistakes: wrong directory, weak password, no backup, nested directories, insecure permissions
  • Encryption is transparent: work in plain directory; encryption/decryption happen automatically
  • Test remounting to verify data persists and decrypts correctly
  • Keep file permissions at 700—don’t make directories world-readable

Daily Mount and Unmount Workflow

  • Daily workflow: Mount at start, unmount at end—build this habit
  • Plain directory must be empty, owned by you, and NOT inside cipher directory
  • Verify mounts: mountpoint -q for scripts, mount | grep gocryptfs for human inspection, plus df and ls
  • Mounted = accessible but protected by FUSE permissions; Unmounted = encrypted on disk
  • Never use allow_other on RESTRICTED data
  • Handle “device busy” with lsof to find processes, then kill and retry
  • Mounts do NOT survive reboots or job completion—this is a security feature
  • Always unmount before leaving your desk or logging out
  • Lazy unmount (fusermount -uz) is last resort when processes won’t close
  • Work transparently in plain directories; encryption happens automatically

Non-Interactive Mounting and Troubleshooting

  • Non-interactive mounting: Use password file (Approach 2) with chmod 600 for SLURM jobs
  • Store password in encrypted /rhome, never in /bigdata or /scratch
  • Three verification methods: mountpoint -q (script-safe), df, and ls
  • Always verify mount succeeded before using encrypted data
  • Always unmount at job end, even if job fails
  • Wrong password = data permanently inaccessible (no recovery)
  • Common errors: directory not empty, wrong password, permission denied, address in use
  • Use lsof to find open files causing “device busy” errors
  • Lazy unmount (fusermount -uz) is last resort
  • Mounts do NOT survive reboots or job completion
  • Implement proper error handling and unmounting in SLURM scripts

SLURM Integration: Decision Framework

  • Pre-mount for interactive work and debugging; mount-in-script for production
  • Never hardcode passwords in job scripts
  • Use password files with chmod 600 for recommended security
  • Understand your workflow type before choosing approach
  • Reproducibility and cleanup are key for production jobs

SLURM Script Templates and Patterns

  • Use complete annotated templates as starting points for your scripts
  • Trap EXIT pattern guarantees cleanup on all exit paths (success, error, timeout, cancellation)
  • Choose storage based on your needs: /tmpfs for speed, /bigdata for persistence
  • Always check mount success before proceeding with analysis
  • Capture exit codes and return original exit codes for proper job status tracking

Advanced SLURM Patterns

  • Use job arrays to run many similar tasks with independent mounts
  • Build multi-step pipelines that maintain encrypted context across steps
  • GPU jobs need module loading and verification before mounting encrypted data
  • Implement all 4 error handling patterns: mount success, data verification, work completion, space checks
  • Log mount success/failure and work progress but never log passwords or decrypted paths
  • Understand SLURM audit trail: admins see job names, scripts, arguments but not encrypted data contents
  • Always include cleanup trap to guarantee unmounting on success, failure, timeout, or cancellation

Key Backup Strategy

  • Two critical components: gocryptfs.conf (the master key) and your password (the decryption key). Losing either = permanent data loss.
  • gocryptfs.conf contains your encrypted master key and scrypt parameters; every field is essential and cannot be recreated.
  • Implement 3-2-1 backup rule: 3 copies of gocryptfs.conf, 2 different media types (network storage + physical USB), 1 offsite location.
  • Store backups in: /rhome (on-site), USB drive (physical offsite), and encrypted cloud (Bitwarden or 1Password).
  • Test backup restoration regularly to verify your backup strategy actually works.
  • Use chmod 600 for backup files to ensure only owner can read them.
  • Never keep backups in insecure locations (sticky notes, unlocked drawers, email as text).

Password Management

  • Never rely on memorization alone for multiple gocryptfs passwords; use a password manager.
  • Bitwarden is the recommended choice: free, open-source, encrypted end-to-end, available on all platforms.
  • Create strong passwords using the diceware method or password manager generation (14+ characters minimum).
  • Only write down passwords in sealed envelopes stored in locked safes, and destroy the written copy after setting up password manager.
  • Never share passwords via email, text, or messaging; use in-person conversation or encrypted channels only.
  • gocryptfs -passwd changes only the password, not the master key; files are not re-encrypted.
  • After changing a password, update backups of gocryptfs.conf and test that the new password works.
  • Change passwords annually or immediately if you suspect compromise or personnel changes.

Lab Management and Disaster Recovery

  • Establish clear procedures for handling encryption when lab members leave (test access, change passwords, backup updates).
  • Create succession plans with named successors and sealed documents in safe locations before they’re needed.
  • Maintain inventory of all encrypted directories with backup locations, access controls, and compliance notes.
  • Implement automated backup scripts and add them to cron for monthly execution.
  • Test backup restoration regularly to verify recovery procedures actually work.
  • Know the five disaster scenarios: (1) have conf + password (recover), (2) lost conf no backup (permanent loss), (3) forgot password (permanent loss), (4) lost conf with backup (recover), (5) storage failure (permanent unless offsite backup).
  • Document compliance requirements (NIST SP 800-171, HIPAA, IRB) alongside encryption procedures.
  • Keep a recovery contact list for technical, security, and compliance issues.
  • Review encryption management and inventory annually with department leadership.
  • Never store passwords unencrypted; use password manager or sealed physical safe only.

Security Best Practices and Troubleshooting

  • Security habits become automatic: unmount at end of session, keep gocryptfs.conf backed up, verify passwords work
  • The 10 essential security habits protect encrypted data across its entire lifecycle
  • Common mistakes (leaving mounted, storing plaintext copies, sharing passwords, hardcoding passwords, not testing recovery) undermine encryption—avoid all of them
  • Troubleshoot systematically using the problem table; “transport endpoint not connected” is usually just a stale mount
  • Performance optimization focuses on smart mount placement (/tmpfs for temporary data) and batching operations

Verification, Sharing, and Maintenance

  • Verify encryption with a complete testing protocol quarterly: can you mount, unmount, remount, use wrong password, restore from backup?
  • Share encrypted data securely by creating separate encrypted directories per person, not sharing passwords
  • Maintain regular schedules: daily unmount, weekly check, monthly verify, quarterly full test, annual disaster recovery drill
  • Encrypt data immediately when it arrives and is classified as restricted; follow your IRB/compliance retention policy
  • Integration with OnDemand, VS Code SSH, and rclone makes encrypted workflows practical and productive
  • Contact immediately for suspected breaches; help is most effective when reported quickly
  • Treat rclone crypt as a separate encryption layer for cloud-based sharing scenarios
  • Document your encrypted directories, backup locations, and retention dates in an inventory file