Summary and Schedule

Callout

Pomona College HPC Workshop Series

This workshop is part of the Pomona College Research Computing Workshop Series, adapted from Automation and Make by Software Carpentry.

Cluster: sagehen.hpc.pomona.edu Web Portal: OnDemand Support:

Make is available on the Sagehen HPC cluster by default. Automating your analysis pipelines with Make is especially valuable for reproducible HPC workflows.

Adapted for Pomona College by Andrew Wilson, ITS Research Computing. Licensed under CC-BY 4.0.

Make is a tool which can run commands to read files, process these files in some way, and write out the processed files. For example, in software development, Make is used to compile source code into executable programs or libraries, but Make can also be used to:

  • run analysis scripts on raw data files to get data files that summarize the raw data;
  • run visualization scripts on data files to produce plots; and to
  • parse and combine text files and plots to create papers.

Make is called a build tool - it builds data files, plots, papers, programs or libraries. It can also update existing files if desired.

Make tracks the dependencies between the files it creates and the files used to create these. If one of the original files (e.g. a data file) is changed, then Make knows to recreate, or update, the files that depend upon this file (e.g. a plot).

There are now many build tools available, all of which are based on the same concepts as Make.

Prerequisite

Prerequisites

In this lesson we use make from the Unix Shell. Some previous experience with using the shell to list directories, create, copy, remove and list files and directories, and run simple scripts is necessary.

Prerequisite

Setup

In order to follow this lesson, you will need to download some files. Please follow instructions on the setup page.

Acknowledgments

Developed by Andrew Wilson, Director of Research Computing and Digital Scholarship at Pomona College, with Andrei Motchenko, who tested, edited and produced screenshots for the workshop series.

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.

This workshop runs equally well on a personal laptop or on Sagehen, Pomona College’s research computing cluster. To use Sagehen:

  1. Sign in at https://ondemand.hpc.pomona.edu using your Pomona credentials and DUO MFA at https://duo.pomona.edu.
  2. Open Clusters > Sagehen Shell Access for a browser-based bash terminal.
  3. make is available on every Sagehen node, so you can build and re-run pipelines directly on the cluster — including from inside a SLURM batch job.
  4. Cecil Sagehen’s lab teammates often keep their pipelines under /bigdata/lab/<labname>/pipelines/ so the Makefile and its outputs are versioned alongside the data.

Files

You need to download some files to follow this lesson:

  1. Download make-lesson.zip.

  2. Move make-lesson.zip into a directory which you can access via your bash shell.

  3. Open a Bash shell window.

  4. Navigate to the directory where you downloaded the file.

  5. Unpack make-lesson.zip:

$ unzip make-lesson.zip
  1. Change into the make-lesson directory:
$ cd make-lesson

Software

You also need to have the following software installed on your computer to follow this lesson:

GNU Make

Linux

Make is a standard tool on most Linux systems and should already be available. Check if you already have Make installed by typing make -v into a terminal.

One exception is Debian, and you should install Make from the terminal using sudo apt-get install make.

OSX

You will need to have Xcode installed (download from the Apple website). Check if you already have Make installed by typing make -v into a terminal.

Windows

Use the Software Carpentry Windows installer.

Python

Python2 or Python3, Numpy and Matplotlib are required. They can be installed separately, but the easiest approach is to install Anaconda which includes all of the necessary python software.