Key Points

What Are Jupyter Notebooks?

  • Jupyter Notebooks combine code, output, and narrative text in interactive documents
  • They’re ideal for exploration, teaching, and communicating research
  • Notebooks are great for development but scripts are better for long-running production jobs
  • The best practice: Start with Jupyter, convert to scripts when ready to scale

Launching Jupyter on Sagehen HPC

  • OnDemand provides a web-based interface for launching Jupyter at https://ondemand.hpc.pomona.edu/
  • Request compute resources (cores, memory, time) when launching
  • Use conservative settings (1 core, 4 GB, 1 hour) for learning
  • The amd partition is the default for standard CPU work

Notebook Interface Basics

  • JupyterLab provides a modern interface with file browser, menus, and tabbed editing
  • Create notebooks by clicking “+” and selecting a Python kernel
  • Your Jupyter session has full access to Sagehen storage (/rhome, /bigdata, /scratch)
  • Always stop your session when done to free cluster resources
  • Your notebooks are saved on the cluster, accessible in future sessions

Working with Cells

  • Notebooks contain code cells (executable) and markdown cells (formatted text)
  • Cells run in the order you execute them, not necessarily top-to-bottom
  • The kernel maintains state (variables, imports) across cells
  • Always arrange cells logically and run them in order
  • Each cell should do one thing; keep cells focused and manageable

Python Fundamentals in Notebooks

  • Python code runs normally in Jupyter cells, with enhanced display capabilities
  • NumPy, pandas, and matplotlib are the core data science libraries
  • pandas can load data from CSV, Excel, JSON, HDF5, and many other formats
  • Magic commands (% and %%) add special IPython functionality like timing code
  • Use print() or display() to show multiple outputs from a single cell

Data Analysis in Notebooks

  • Always inspect shape, dtypes, and null counts as the first cell after loading
  • Use .loc for label-based selection and .iloc for position-based; mixing them is a common bug
  • Boolean filters with multiple conditions need parentheses: df[(a) & (b)]
  • For files over 2 GB, do exploratory work in an interactive SLURM allocation, not on the login node
  • Save final results to /bigdata, not /scratch (which is wiped without backups)
  • Notebook hygiene: every modification cell should produce verifiable output

Visualization in Notebooks

  • Jupyter displays matplotlib plots inline below code cells
  • Line plots, scatter plots, histograms, and bar charts cover most visualization needs
  • Use figsize to control plot dimensions and tight_layout() to prevent overlap
  • Subplots let you compare multiple visualizations in a single figure
  • Always add titles, axis labels, and legends for clarity

Managing Kernels and Environments

  • The kernel is the Python interpreter that maintains all variables and state
  • Restart the kernel to clear state and verify notebooks work from scratch
  • Interrupt the kernel to stop long-running cells
  • Python environments isolate packages, preventing version conflicts
  • conda is Pomona’s recommended tool for managing environments
  • Install kernels to use custom environments in Jupyter notebooks

Sharing and Exporting Notebooks

  • Convert notebooks to Python scripts for batch jobs, HTML for sharing, PDF for publication
  • Export environment.yml files so others can recreate your exact environment
  • Use Git for version control; commit notebooks, environment.yml, and documentation
  • Publish on GitHub and use Zenodo for citable DOIs
  • Always restart kernel and run all cells before sharing

Best Practices

  • Organize projects with clear directory structure and naming conventions
  • Document each notebook with purpose, author, data sources, and methods
  • Scale from Jupyter to batch jobs: develop interactively, then convert to scripts
  • Avoid common mistakes: kernel state issues, relative paths, outdated packages
  • Always export environment.yml for reproducibility
  • Test reproducibility: Restart kernel and run all cells before sharing