Key Points

Introduction to GPU Computing

  • GPUs excel at parallel computation for thousands of operations
  • Sagehen has 10 GPUs across three types: A100, L40S, and RTX PRO 6000
  • Deep learning, simulations, and data processing benefit most
  • Framework libraries (PyTorch, TensorFlow) handle GPU automatically
  • Not all workloads benefit; profile before investing GPU time

Sagehen HPC GPU Hardware Configuration

  • Sagehen has 10 GPUs across three types: A100, L40S, and RTX PRO 6000
  • RTX PRO 6000: Largest memory on Sagehen (96 GB GDDR7 ECC, Blackwell)
  • A100: 80 GB HBM2e, best training throughput for large models
  • L40S: Best value for most deep learning at 48 GB
  • Use –gres=gpu:type:count to request specific GPUs
  • Monitor memory usage; request right-sized GPU
  • Consider cost-benefit and queue depth when choosing GPU type

Requesting and Allocating GPUs

  • GPU jobs require: –partition=gpu and –gres=gpu:N
  • Request matching CPUs and memory for your GPU
  • Max 720 hours per job; typical 1-8 hours
  • Interactive sessions available for testing
  • Multiple GPUs require distributed training setup

GPU Software Setup and Modules

  • Load GPU modules: pytorch, tensorflow, cuda
  • CUDA toolkit handles GPU computation
  • Create conda environments for reproducibility
  • Verify GPU availability before running jobs
  • Test GPU setup with simple benchmark script

Training Models with PyTorch and TensorFlow

  • Move models to GPU: model.cuda()
  • Move data to GPU: data.cuda()
  • TensorFlow automatically uses GPU
  • Monitor with nvidia-smi
  • Optimize batch size for memory
  • Use gradient checkpointing for large models
  • Consider mixed precision for speed

Monitoring and Optimization

  • Use nvidia-smi to monitor GPU utilization and memory
  • Aim for 80-100% GPU utilization
  • Profile code to identify bottlenecks
  • Batch size, data loading, and precision affect performance
  • Monitor memory to avoid OOM errors

GPU Etiquette and Best Practices

  • Sagehen HPC has limited GPUs; fair sharing is essential
  • Request appropriate GPU type for your workload
  • Don’t monopolize GPUs or leave idle
  • Test on L40S before scaling to A100
  • Clean up after jobs complete
  • Monitor your usage with squeue
  • Respect other users’ resource needs