Key Points

Why Software Modules?

  • Environment modules solve the problem of multiple software versions on shared systems
  • Modules temporarily modify your shell environment (PATH, LD_LIBRARY_PATH, etc.)
  • No files are installed or removed; only your environment changes
  • Modules follow the naming convention software/version
  • Modules with (D) are defaults; specify versions explicitly to get other versions

Finding and Loading Modules

  • Use module avail with search terms or grep to find software
  • module load activates software; module unload deactivates it
  • Use module help to understand what a module does before loading
  • Use module show to see exact environment modifications
  • Modules can automatically load prerequisites
  • If software isn’t available, contact to request it

Module Collections and Defaults

  • Load multiple modules with module load module1 module2 module3
  • Use module swap to switch versions cleanly
  • Use module purge to start fresh
  • Save configurations with module save <name> and restore with module restore <name>
  • Set module save default to auto-load modules on login
  • Always load modules in job scripts; don’t rely on your login environment

Introduction to Conda

  • Modules load system software; conda manages Python packages in isolated environments
  • Create environments with conda create -n name python=version
  • Use conda activate to enter an environment; conda deactivate to exit
  • Most workflows use both modules and conda together
  • Never install into the base environment on shared systems

Creating Conda Environments

  • Install packages with conda install package1 package2
  • Use conda list to see what’s installed and check versions
  • Use -c conda-forge for packages on the conda-forge channel
  • Create one environment per project to prevent dependency conflicts
  • Always load miniconda3 and activate your environment in job scripts

Managing Conda Packages

  • Export with conda env export > environment.yml for reproducibility
  • Recreate with conda env create -f environment.yml
  • Use --from-history for cross-platform environment files
  • Monitor disk usage with du -sh ~/.conda/envs/*
  • Remove old environments with conda env remove -n name
  • Clean conda cache with conda clean --all to free space

pip and Virtual Environments

  • Use conda for scientific computing with complex dependencies
  • Use pip for pure Python packages or when adding to conda environments
  • Use venv for lightweight, fast Python isolation
  • Export with pip freeze > requirements.txt for pip-based reproducibility
  • For shared systems, avoid pip install --user and use conda or venv instead

Combining Conda and pip

  • Combine conda (for scientific packages) and pip (for pure Python packages) in one environment
  • The exported environment.yml captures both conda and pip packages
  • Use conda for the base, then pip for packages not available in conda
  • Both venv/pip and conda approaches work in SLURM job scripts
  • Always export your environment for reproducibility regardless of which tools you use

Environment Reproducibility

  • Always load modules and activate environments explicitly in job scripts
  • Start job scripts with module purge for a clean environment
  • Pin package versions in environment.yml for reproducibility
  • Document software choices in version control
  • Create one environment per project with its own environment.yml
  • Update pinned files carefully and commit changes to git

Best Practices and Troubleshooting

  • Monitor disk usage regularly and clean up old conda environments
  • Test scripts locally and with small jobs before submitting large ones
  • Contact with job ID, error message, and what you’ve tried
  • Follow the new project checklist for reproducible research setups
  • Use module purge at the start of job scripts for a clean environment
  • Start with module purge, then load specific module versions explicitly