Key Points

Why Parallel Computing?

  • Parallel computing trades computational resources to reduce wall clock time
  • Speedup and efficiency quantify parallel performance
  • Profile and measure before parallelizing
  • Not all problems benefit from parallelism

Amdahl's Law and Speedup

  • Amdahl’s Law shows that serial code limits maximum speedup
  • Even small serial fractions (5-10%) become significant bottlenecks at scale
  • Profile serial code first to identify the true bottleneck
  • Don’t parallelize prematurely; ensure the payoff justifies the complexity

Types of Parallelism

  • Shared-memory parallelism works within a node; distributed-memory across nodes
  • Data parallelism applies the same operation to different data
  • Task parallelism runs different tasks in parallel
  • Most research problems are embarrassingly parallel at some level

Embarrassingly Parallel Problems

  • Embarrassingly parallel problems have no dependencies between tasks
  • Common patterns: parameter sweeps, Monte Carlo, batch processing, resampling
  • Convert serial code by accepting parameters and using unique output files
  • These are the easiest to parallelize and often the most valuable

GNU Parallel and Shell Parallelism

  • parallel -j N runs up to N tasks concurrently with throttling, logging, and resume
  • Background jobs (&) work for trivial cases but lack concurrency limits and per-task logs
  • --joblog plus --resume-failed makes long batches restartable
  • --tag and per-task log files keep output legible
  • Structure scripts to take input as args and write unique output files
  • Stage shared input on /scratch to avoid hammering /bigdata with parallel reads
  • Move from shell parallelism to SLURM job arrays when tasks exceed one node or run for many hours

SLURM Job Arrays

  • SLURM job arrays submit many similar jobs with one command
  • Each task gets a unique SLURM_ARRAY_TASK_ID
  • Map array indices to parameters and data files
  • Save results with unique filenames based on task ID
  • Monitor and manage entire arrays with a single job ID
  • Use --array=1-1000%50 to throttle concurrency on shared clusters
  • Aim for tasks of at least 5 minutes; batch internally if shorter
  • scontrol requeue re-runs only failed tasks without redoing the rest

Advanced Job Arrays

  • Map single array indices to multi-parameter grids using division and modulo
  • Parameter files provide a clean way to define complex sweeps and version-control them
  • Use argparse so the work script is independent of the SLURM script
  • Batch small tasks into fewer jobs to reduce scheduling overhead
  • Use sacct to review completed array performance and find failures
  • Submit an aggregation job with --dependency=afterok:$SWEEP_ID
  • Throttle concurrency with --array=1-1000%20 on busy clusters

Shared Memory with OpenMP

  • In Python, use multiprocessing (not threading) for true parallelism on CPU-bound work
  • Use threading for I/O-bound work (downloads, database calls)
  • Request cores with #SBATCH --cpus-per-task=N
  • Read os.environ['SLURM_CPUS_PER_TASK'] so scripts adapt to allocation
  • Use Pool.map() to parallelize independent tasks
  • Avoid race conditions by not sharing mutable state; use map-reduce
  • Sagehen amd nodes have 128 cores and 512 GB; a single node is often enough
  • Shared-memory parallelism is limited to a single node; arrays scale further

Threading in Python and R Parallelism

  • Threading is right for I/O-bound work in Python; multiprocessing for CPU-bound
  • R’s parallel package provides parLapply for multi-core computation
  • Use clusterExport to share function definitions with R workers
  • Common patterns: file processing, bootstrap resampling, parameter sweeps
  • Match process count to SLURM_CPUS_PER_TASK, not the node’s physical cores
  • Avoid pickling large data; use memory-mapped files for shared inputs
  • Profile before parallelizing; the gain depends on what dominates runtime

Distributed Computing Concepts

  • Distributed-memory systems have separate memory per node
  • MPI enables explicit message passing between processes
  • Use MPI only for tightly-coupled algorithms across multiple nodes
  • Launch MPI with srun under SLURM
  • InfiniBand makes MPI practical on Sagehen

MPI Basics

  • mpi4py provides Python bindings for MPI
  • Each rank knows its own ID (rank) and total count (size)
  • scatter distributes data, reduce combines results
  • Use srun (not mpirun) on Sagehen to launch MPI jobs correctly
  • Each rank works on its portion independently, then results are gathered
  • Strong scaling has limits (Amdahl’s law); weak scaling is more achievable
  • MPI is powerful but complex; reserve for truly distributed problems
  • For deep learning, prefer DistributedDataParallel over raw MPI

Choosing the Right Approach

  • Follow a decision framework: measure, profile, analyze, choose, verify
  • Job arrays are the simplest approach for embarrassingly parallel problems
  • Profile serial code first to identify bottlenecks
  • Don’t over-engineer: use the simplest approach that works
  • Always validate that parallelization actually improves performance