Image 1 of 1: ‘Two panels. On the left, serial execution: Task A, Task B and Task C each take two hours and run one after another, giving a total of six hours. On the right, parallel execution: the same three tasks run at the same time and all feed into a total of two hours, with a note that the job finishes when the slowest task does.’
Three independent two-hour tasks: run one after
another and the job takes six hours; run them at the same time and it
takes two.
Image 1 of 1: ‘Two panels illustrating Amdahl's Law. On the left, code is split into a serial portion of twenty percent that cannot be parallelised and a parallel portion of eighty percent that can use many cores. On the right, total time as cores increase: one core takes one hundred percent, four cores take twenty plus eighty over four equals forty percent, sixteen cores take twenty plus eighty over sixteen equals twenty-five percent, and infinite cores still take twenty percent because the serial portion remains.’
The serial fraction sets a floor no amount of
hardware can get below.
Image 1 of 1: ‘Two panels. Shared memory on one node shows four cores all reading the same 500 GB of RAM, used by OpenMP, threading and multiprocessing, with a note that all 128 cores on a node share that memory. Distributed memory across many nodes shows Node A and Node B, each with 128 cores and their own RAM, connected through a network, with a note that data must be sent explicitly using MPI.’
Shared memory keeps everything in one address
space; distributed memory makes you send data across the network
yourself.
Image 1 of 1: ‘A decision tree starting from the question, is my problem embarrassingly parallel. If the tasks are not independent, the answer is that it is not embarrassingly parallel and MPI is covered in episode 11. If they are independent, the next question is whether results can be combined afterwards; if yes, use a SLURM job array with sbatch --array=1-100, and if no, restructure the problem first.’
If the tasks are independent and the results can
be combined afterwards, a job array is all you need.
Image 1 of 1: ‘Terminal on Sagehen HPC. The command module avail parallel returns No module(s) or extension(s) found, followed by the usual Lmod suggestions to try module --default avail or module spider.’
Checked on Sagehen HPC: there is no
parallel module, which is why the episode installs it into
your own conda environment.
Image 1 of 1: ‘A flow diagram. The command sbatch --array=1-100 my_script.sh causes SLURM to create 100 independent jobs, which run in parallel. Job 1 has SLURM_ARRAY_TASK_ID equal to 1, Job 2 has 2, and Job 100 has 100. All of them feed into a step that combines the results. A caption notes that each job reads its own SLURM_ARRAY_TASK_ID and works on that slice of the problem.’
One sbatch command creates a hundred independent
jobs, each told which slice of the work it owns.
Image 1 of 1: ‘A five-step flow. Define the parameters, for example learning rate values of 0.001, 0.01 and 0.1 and batch sizes of 32, 64 and 128. Write them to a parameter file with nine combinations, one per line. Submit one array job with sbatch --array=1-9 sweep.sh. Each task reads the line matching its SLURM_ARRAY_TASK_ID and runs that experiment. Finally, collect the results and pick the best.’
A parameter sweep is a job array over a list of
settings, one line per combination.
Image 1 of 1: ‘Four panels showing MPI collective operations. Reduce sums values 3, 7 and 2 from ranks 1 to 3 into 12 on rank 0. Gather collects B, C and D from ranks 1 to 3 into A B C D on rank 0. Scatter sends the parts of A B C D from rank 0 out to ranks 1, 2 and 3. Broadcast copies the value A from rank 0 to all three other ranks.’
The four collective operations, and which
direction the data moves in each.
Image 1 of 1: ‘A decision tree for choosing a parallelisation strategy. If the tasks are fully independent, start with SLURM job arrays. If not, ask whether the work fits on one node of up to 128 cores; if it does, use shared memory with OpenMP or multiprocessing. If not, ask whether it must span multiple nodes; if it must, use MPI, the most complex option, and if not, rethink the problem.’
Work down the list and stop at the first
approach that fits.
