Chapter 15: HPC Cluster Usage — modules, SLURM, jobs¶

Source slides: V13_HPC_cluster.pdf (final lecture). The lecture references the Phoenix cluster (TU Braunschweig) and the LUIS HPC at Hannover as concrete examples.

1. Chapter Overview¶

The HPC cluster is the machine on which you run all the things you've learned so far: the build process, MPI/OpenMP code, gnuplot post-processing, Git pulls. The cluster has a special set of conventions:

Architecture: login nodes (where you SSH), compute nodes (only via the scheduler), GPU nodes, fat nodes, visualization nodes, storage.
Modules (module load, module list, module spider) — pick compiler/MPI versions on demand.
SLURM workload manager — sbatch, squeue, srun, salloc, sinfo, scancel.
Job script (run.sbatch) with SBATCH directives (nodes, ntasks, time, partition).
Storage tiers — $HOME slow but persistent, $WORK/$SCRATCH fast but periodically purged, $ARCHIVE tape.
Queues / partitions — different node types have different limits.

Why it matters in HPC/CFD: you cannot run a CFD job by SSH-ing and typing mpirun on the login node — it would be killed and you'd be banned. You must learn to write a job script.

What the examiner asks (very common):

"What is a login node? Why not run heavy jobs on it?"
"What is a module?"
"Write a SLURM batch script for an MPI job using 4 nodes × 16 tasks for 2 hours."
"Difference between srun, sbatch, salloc."
"Explain sinfo/squeue/scancel."
"What's a partition?"

What you must master for top grade:

The cluster topology diagram (login / compute / storage / VPN).
Module commands.
A complete run.sbatch for an MPI+OpenMP CFD job.
SLURM commands with their roles.
Storage best practices.

2. Basics from Zero¶

A cluster is many Linux servers ("nodes") tied together by a fast network and shared filesystem. You connect via SSH only to the login node. The login node is shared by all users for compiling, editing, copying — never for running real simulations. To compute, you submit a job to the scheduler (SLURM); the scheduler allocates one or more compute nodes and runs your program there.

বাংলায়: Cluster-এ ঢোকার একটাই দরজা: login node — এখানে শুধু edit, compile, ফাইল কপি করা যায়, simulation চালানো কঠোরভাবে নিষেধ, কারণ এটা শত শত user-এর shared জায়গা। আসল হিসাব চলে compute node-এ, আর সেখানে পৌঁছানোর একমাত্র রাস্তা হলো SLURM scheduler-এর কাছে job জমা দেওয়া। পরীক্ষার এক নম্বরের প্রশ্ন: "কেন login node-এ mpirun চালানো যাবে না?" — উত্তর: shared resource, scheduler-ই compute node ন্যায্যভাবে বরাদ্দ করে।

Software on the cluster is provided as modules. Instead of having one fixed compiler version, you load what you need:

module avail
module load gcc/12 openmpi/4.1 cmake/3.27
module list

Loading a module sets PATH, LD_LIBRARY_PATH, MANPATH, etc. The benefit: many users can use different versions side by side.

বাংলায়: Module system-টা আসলে environment variable-এর খেলা: module load করলে PATH আর LD_LIBRARY_PATH-এ ওই software-এর directory ঢুকে যায়, unload করলে বেরিয়ে যায় — কিছুই install বা uninstall হয় না। এতেই এক cluster-এ একজন gcc/12 আর আরেকজন gcc/13 পাশাপাশি চালাতে পারে। মনে রাখো: sbatch-এ জমা দেওয়া job তোমার shell-এর module পায় না, তাই script-এর ভেতরে আবার module load লিখতেই হবে — এটা পরীক্ষার খুব প্রিয় trap।

A batch script is a small shell script with #SBATCH … headers describing what you want, and shell commands describing what to run:

#!/bin/bash
#SBATCH --job-name=cfd_test
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16
#SBATCH --time=02:00:00
#SBATCH --partition=standard
#SBATCH --output=run_%j.log
module load gcc/12 openmpi/4.1
mpirun -n $SLURM_NTASKS ./solver in.dat

Submit with sbatch run.sbatch. SLURM returns a job ID. Watch with squeue -u $USER. Cancel with scancel <jobid>.

বাংলায়: Job script-এর গঠনটা দুই ভাগে: উপরে #SBATCH লাইনগুলো হলো scheduler-এর কাছে অনুরোধ (কয়টা node, কত সময়, কোন partition), আর নিচের সাধারণ shell command-গুলো হলো node পাওয়ার পরে আসলে কী চালাবে। sbatch দিলে সাথে সাথে একটা job ID ফেরত আসে, কিন্তু job তখনই চলে না — আগে queue-তে অপেক্ষা (PENDING), তারপর resource খালি হলে RUNNING। squeue, scancel, sacct — এই তিনটা observation command পরীক্ষায় প্রায় নিশ্চিত আসে।

Real-life analogy. The cluster is an airport. The login node is the check-in counter (everyone meets here briefly). Compute nodes are the planes — you don't board them directly; you give SLURM your ticket (job script) and it finds a plane.

Real-life HPC use. TU Braunschweig's Phoenix: 304 compute nodes, 8 fat, 8 GPU, 2 login. You SSH in, edit / build, then sbatch your CFD case.

What if you misunderstand? Running mpirun -n 64 on the login node — kills the login session for everyone, you get warning emails, your account may be suspended. Or you put outputs in $HOME (limited quota) and fill it.

3. Hard English Made Easy¶

Hard Term	Simple English	বাংলা	Example
Login node	Entry node, no heavy jobs	লগইন নোড	`ssh user@phoenix`
Compute node	Where jobs run	কম্পিউট নোড	node001
GPU node	Compute node with GPUs	জিপিইউ নোড	gpu01
Fat node	Lots of RAM	বিশাল RAM নোড	fat01
Visualization node	Used for ParaView etc.	ভিজ নোড	vis01
Storage node	Disks/file system	স্টোরেজ নোড	NFS/Lustre
Lustre / GPFS	Parallel filesystem	প্যারালেল ফাইল সিস্টেম	`$WORK`
Module	Software package	সফটওয়্যার মডিউল	`module load`
Lmod	Modern module system	মডার্ন মডিউল	`module spider`
Workload manager	Job scheduler	জব শিডিউলার	SLURM
SLURM	Simple Linux Utility for Resource Management	SLURM	`sbatch`
Partition	Group of similar nodes	নোড গ্রুপ	`standard`
Queue	Waiting list	অপেক্ষমান তালিকা	`squeue`
Job script	Batch script	ব্যাচ স্ক্রিপ্ট	`run.sbatch`
sbatch directive	`#SBATCH` line	sbatch নির্দেশ	`#SBATCH --time=1:00:00`
Job ID	SLURM ID	জব আইডি	`12345`
Walltime	Maximum runtime allowed	সর্বোচ্চ সময়	`02:00:00`
Allocation	Resources granted	বরাদ্দ	nodes+cores+RAM
Interactive job	Shell on a compute node	ইন্টারঅ্যাকটিভ সেশন	`srun --pty bash`
Quota	Storage limit	কোটা	`$HOME 100 GB`
Scratch	Fast temp storage	দ্রুত অস্থায়ী স্টোরেজ	`$WORK`
Archive	Long-term storage	দীর্ঘমেয়াদী আর্কাইভ	tape
VPN	Remote-network gateway	ভিপিএন	needed off-campus

4. Deep Theory Explanation¶

4.1 Cluster topology (slide 8)¶

 USER (laptop)
   │  ssh (through VPN if off-campus)
   ▼
┌──────────────────┐   edit, compile, sbatch — NEVER mpirun here
│   LOGIN NODE(S)  │◄─ shared by all users, small CPU/RAM
└────────┬─────────┘
         │ sbatch run.sbatch  (job enters the queue)
         ▼
┌──────────────────┐   decides WHO runs WHERE and WHEN;
│ SLURM SCHEDULER  │◄─ priorities, partitions, backfill
└────────┬─────────┘
         │ allocates nodes, starts the job script
         ▼
┌─────────────────────────────────────────────────────┐
│                COMPUTE PARTITIONS                   │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐    ┌─────────┐ │
│ │ node001 │ │ node002 │ │ node003 │ ·· │ gpu/fat │ │
│ └────┬────┘ └────┬────┘ └────┬────┘    └────┬────┘ │
│      └───────────┴─────┬─────┴──────────────┘      │
│             fast interconnect (InfiniBand)         │
└────────────────────────┬────────────────────────────┘
                         │ every node mounts the same FS
                         ▼
        ┌────────────────────────────────┐
        │  PARALLEL FILESYSTEM (Lustre)  │
        │  $HOME (small, backed up)      │
        │  $WORK (big, fast, purged)     │──► $ARCHIVE (tape)
        └────────────────────────────────┘

Login nodes have small CPU/RAM; compute nodes have much more. The shared storage is mounted on every node so a file you create from the login node is visible on the compute node.

বাংলায়: পুরো cluster-এর গল্পটা এই এক ছবিতে: user ssh দিয়ে শুধু login node পর্যন্ত যায়, সেখান থেকে sbatch দিয়ে job-টা SLURM scheduler-এর হাতে তুলে দেয়, আর scheduler-ই ঠিক করে কোন compute node-এ কখন চলবে। নিচের parallel filesystem (Lustre) সব node-এ mount করা থাকে — তাই login node-এ লেখা ফাইল compute node থেকেও দেখা যায়, আলাদা করে কপি করতে হয় না। পরীক্ষায় architecture আঁকতে বললে এই চারটা স্তর (user → login → scheduler → compute) আর পাশে storage দেখালেই হবে।

4.2 Cluster nodes by type (slide 9)¶

Type	Purpose
Login	SSH access, edit/compile, light tests
Compute	Default jobs
GPU	Jobs requiring CUDA/HIP
Fat	Large-RAM jobs (>1 TB)
Visualization	Remote ParaView/VNC
Storage	Holds data
Managing	Monitoring/scheduling

4.3 Storage tiers (slide 11)¶

$HOME — small (e.g. 100 GB), backed up, slow.
$WORK / $SCRATCH — large, fast, parallel filesystem (Lustre / GPFS), periodically purged (e.g. files older than 14 days).
$ARCHIVE — tape, very slow, very large.

Rule of thumb: build & run in $WORK; keep code in $HOME or git; archive results in $ARCHIVE.

বাংলায়: তিনটা storage-এর চরিত্র তিন রকম: HOME ছোট কিন্তু backup হয় (code রাখো), WORK বিশাল আর দ্রুত কিন্তু পুরনো ফাইল মুছে ফেলা হয় (simulation-এর input/output এখানেই), আর ARCHIVE হলো tape — বছর ধরে রাখা যায় কিন্তু পড়তে ঘণ্টা লাগতে পারে। সবচেয়ে দামি নিয়ম: বড় I/O কখনো HOME-এ নয়, কারণ quota ভরে গেলে নিজের (এবং অন্যদের) কাজ আটকে যায়। পরীক্ষায় "কোথায় কী রাখবে" প্রশ্নে এই তিন স্তরের নাম-বৈশিষ্ট্য-উদাহরণ লিখে দাও।

4.4 Modules¶

module is a meta-tool for switching environment. It uses Lmod (modern Tcl-based) on most clusters.

Command	Effect
`module avail`	list available
`module list`	currently loaded
`module spider`	search (Lmod)
`module load gcc/12`	load
`module unload gcc/12`	unload
`module switch gcc/12 gcc/13`	swap
`module purge`	unload all
`module show gcc/12`	what it sets
`module help`	help

Loading a module typically appends to PATH and sets LD_LIBRARY_PATH/LIBRARY_PATH/CPATH so the compiler and linker find the right binaries/headers/libs.

বাংলায়: এই টেবিল থেকে পরীক্ষার জন্য পাঁচটা command যথেষ্ট: avail (কী কী আছে), load/unload, list (এখন কী চালু), spider (খোঁজা), purge (সব খুলে ফেলা)। purge দিয়ে শুরু করে দরকারিগুলো load করা হলো reproducible environment-এর সোনার নিয়ম — job script-এ এই জোড়াটাই লেখা উচিত। module show দিয়ে দেখা যায় একটা module আসলে কোন কোন variable বদলায়, "module কীভাবে কাজ করে" প্রশ্নে এটা উল্লেখ করলে বাড়তি নম্বর।

4.5 SLURM commands¶

Command	Purpose
`sinfo`	partition / node availability
`squeue`	job queue (`-u $USER` for yours only)
`sbatch script.sh`	submit batch job
`srun [opts] cmd`	run interactively / step inside batch
`salloc [opts]`	request allocation, then drop into shell
`scancel <jobid>`	cancel
`scontrol show job <jobid>`	details
`sacct -j <jobid>`	accounting

SLURM JOB STATE LIFECYCLE — which command shows/causes each transition

 $ sbatch run.sbatch
 "Submitted batch job 12345"
        │
        ▼
 ┌──────────────┐  visible in squeue as "PD"; column REASON says why:
 │   PENDING    │  (Priority)  = others are ahead of you
 │              │  (Resources) = not enough free nodes yet
 └──────┬───────┘  (the bigger the request, the longer the wait)
        │  scheduler finds/backfills free nodes
        ▼
 ┌──────────────┐  squeue shows "R" + elapsed time + NODELIST
 │   RUNNING    │  job script executes on the compute node(s)
 └──┬───┬───┬───┘
    │   │   │
    │   │   └── walltime limit hit ──────────► TIMEOUT   (killed by SLURM)
    │   └────── script exits non-zero ───────► FAILED
    │           (or node crash → NODE_FAIL)
    └────────── script exits 0 ──────────────► COMPLETED
        ▲
        │  at ANY point: scancel 12345 ──────► CANCELLED
        │
 After the job leaves the queue, squeue shows NOTHING —
 use  sacct -j 12345 --format=JobID,State,Elapsed,MaxRSS
 to see the final state and resource usage.

বাংলায়: Job-এর জীবনচক্রটা মুখস্থ রাখো: sbatch → PENDING → RUNNING → COMPLETED/FAILED/TIMEOUT, আর যেকোনো সময় scancel দিলে CANCELLED। চলমান অবস্থা দেখায় squeue, কিন্তু job শেষ হয়ে গেলে squeue-তে আর কিছুই দেখা যায় না — তখন ইতিহাস দেখার tool হলো sacct। PENDING-এর পাশে (Priority) মানে সামনে লাইন আছে, (Resources) মানে এখনো যথেষ্ট node খালি নেই — এই দুটো reason-এর পার্থক্য পরীক্ষায় জিজ্ঞেস করার মতো সূক্ষ্ম জিনিস।

4.6 Anatomy of a SLURM batch script¶

#!/bin/bash
#SBATCH --job-name=cfd_test          # name
#SBATCH --output=cfd_%j.log          # stdout
#SBATCH --error=cfd_%j.err           # stderr
#SBATCH --partition=standard         # queue
#SBATCH --nodes=4                    # nodes
#SBATCH --ntasks=64                  # total MPI ranks
#SBATCH --ntasks-per-node=16         # ranks per node
#SBATCH --cpus-per-task=2            # OpenMP threads per rank
#SBATCH --mem-per-cpu=4G             # memory per core
#SBATCH --time=02:00:00              # HH:MM:SS walltime
#SBATCH --mail-type=END              # email at finish
#SBATCH --mail-user=me@uni.de

module purge
module load gcc/12 openmpi/4.1

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
cd $SLURM_SUBMIT_DIR

mpirun -n $SLURM_NTASKS ./solver in.dat

SLURM environment variables inside the job: SLURM_JOB_ID, SLURM_NTASKS, SLURM_NTASKS_PER_NODE, SLURM_CPUS_PER_TASK, SLURM_NNODES, SLURM_SUBMIT_DIR, SLURM_NODELIST.

SBATCH SCRIPT ANATOMY — every line labeled with what it requests

#!/bin/bash ─────────────────────► interpreter; MUST be line 1
#SBATCH --job-name=cfd_test ─────► label shown in squeue
#SBATCH --output=cfd_%j.log ─────► stdout file; %j = job ID
#SBATCH --error=cfd_%j.err ──────► stderr file (separate)
#SBATCH --partition=standard ────► which node group / queue
#SBATCH --nodes=4 ───────────────► number of MACHINES
#SBATCH --ntasks=64 ─────────────► total MPI RANKS (processes)
#SBATCH --ntasks-per-node=16 ────► ranks per machine (4·16=64, consistent)
#SBATCH --cpus-per-task=2 ───────► CORES per rank → OpenMP threads
#SBATCH --mem-per-cpu=4G ────────► RAM per core (64·2·4G = 512G total)
#SBATCH --time=02:00:00 ─────────► WALLTIME limit HH:MM:SS — job is
│                                   KILLED when it expires
│  ┄┄┄ everything above = REQUEST to the scheduler ┄┄┄
│  ┄┄┄ everything below = what RUNS on the node    ┄┄┄
module purge ────────────────────► reproducible clean environment
module load gcc/12 openmpi/4.1 ──► job does NOT inherit your shell's
│                                   modules — must load here!
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK ─► sync threads to request
cd $SLURM_SUBMIT_DIR ────────────► run where the script was submitted
mpirun -n $SLURM_NTASKS ./solver ─► ranks count synced via variable,
                                    never hard-coded

বাংলায়: Script-টার দুই অংশ আলাদা চোখে দেখো: #SBATCH লাইনগুলো sbatch জমা দেওয়ার মুহূর্তে scheduler পড়ে (এগুলো bash-এর কাছে নিছক comment), আর নিচের command-গুলো চলে অনেক পরে, যখন node বরাদ্দ হয়। ভেতরের সংখ্যাগুলোর সামঞ্জস্য পরীক্ষক খুঁটিয়ে দেখে: nodes × ntasks-per-node = ntasks, আর cpus-per-task = OMP_NUM_THREADS। mpirun -n-এ সংখ্যা hard-code না করে $SLURM_NTASKS লেখাটাই professional উত্তর — request বদলালে script নিজে নিজেই ঠিক থাকে।

4.7 Interactive session¶

srun --partition=standard --nodes=1 --ntasks-per-node=4 --time=1:00:00 --pty bash
# or:
salloc --nodes=1 --ntasks-per-node=4 --time=1:00:00
ssh $(squeue -h -j $SLURM_JOB_ID -O nodelist | head -1 | cut -d, -f1)

Useful for debugging and short tests.

4.8 Choosing partition / time / size¶

Test partition (short walltime) for debug.
Standard partition for production.
GPU / fat partition for special hardware.
Walltime: SLURM kills the job at the limit. Start with conservative estimate; check with sacct.

4.9 Diagrams from V13¶

Cluster architecture (slide 8) — see §4.1.
SLURM scheduling (slide 21) — pending → running → completed.
Storage tiers diagram (slide 11).

4.10 Compute-budget arithmetic — core-hours, queues, walltime¶

Core-hour accounting. Clusters bill by reserved resources, not by useful work:

\[ \text{core-hours} = N_{\text{nodes}} \times \text{cores per node} \times \text{walltime in hours}. \]

Worked example. A job on 4 nodes with 48 cores each running 6 h costs

\[ 4 \times 48 \times 6 = 1152 \text{ core-hours}, \]

and this is charged even if your code only used one core per node — you reserved the whole nodes. With a monthly project budget of $50\,000$ core-h you can afford $\lfloor 50\,000 / 1152 \rfloor = 43$ such runs.

Budgeting a strong-scaling study. You measure scaling at $N = 1, 2, 4, 8, 16, 32, 64$ nodes (48 cores each); the baseline run takes $T_1 = 8$ h on one node. With ideal scaling, $T_N = T_1/N$, so every run costs the same:

\[ \text{cost}(N) = N \times 48 \times \frac{8}{N} = 384 \text{ core-h} \quad\Rightarrow\quad \text{total} = 7 \times 384 = 2688 \text{ core-h}. \]

With realistic scaling the cost per run is inflated by the lost efficiency: $\text{cost}(N) = 384 / E(N)$. Taking Amdahl with $f = 0.05$ applied to the node count:

$N$ nodes	$S(N)$	$T_N = 8/S$ (h)	$E(N)$	core-h $= N \cdot 48 \cdot T_N$
1	1.00	8.000	1.000	384.0
2	1.90	4.200	0.952	403.2
4	3.48	2.300	0.870	441.6
8	5.93	1.350	0.741	518.4
16	9.14	0.875	0.571	672.0
32	12.55	0.638	0.392	979.2
64	15.42	0.519	0.241	1593.6

Total: $4992$ core-h — nearly double the ideal 2688, and the single 64-node point alone costs 4× the baseline run. Lesson: scaling studies are paid for in efficiency loss; budget them before submitting.

Why huge jobs queue longer (backfill intuition). The scheduler plays Tetris with nodes and time. A 1-node 1-h job fits into almost any idle gap left in the schedule, so backfilling slots it in immediately (it can jump the queue as long as it does not delay the highest-priority job). A 64-node job can only start when 64 nodes are free simultaneously — the scheduler must deliberately drain nodes (hold them idle as others finish) to assemble the block, and every drained node-hour is wasted capacity. Rough intuition: the chance that $k$ nodes are simultaneously free decays rapidly with $k$, so expected queue wait grows steeply with both node count and requested walltime. Practical corollary: a shorter, honest --time request lets the backfiller place your job into gaps → it starts sooner.

Walltime safety margin. If a pilot run measures $T_{\text{est}} = 5$ h, request roughly

\[ T_{\text{wall}} \approx 1.25 \times T_{\text{est}} = 6.25 \text{ h} \;\Rightarrow\; \texttt{--time=06:15:00}. \]

Reasoning for the asymmetry of the two failure modes: too short → SLURM kills the job at the limit (state TIMEOUT) and you lose the run; too long → the scheduler struggles to find a slot and the job queues far longer. 25% margin is the practical compromise.

বাংলায়: core-hour-এর হিসাব সব ক্লাস্টারের মুদ্রা: nodes × cores × ঘণ্টা। walltime চাওয়ার কৌশলটাও মনে রাখো — মেপে নাও (pilot run), তারপর ~২৫% margin; বেশি চাইলে queue-তে পচবে, কম চাইলে TIMEOUT-এ মরবে। পরীক্ষায় এই trade-off-এর যুক্তিটা লিখতে বলা হয়।

4.11 SLURM job-state lifecycle¶

                 sbatch run.sbatch
                        │
                        ▼
                 ┌──────────────┐   scheduler finds nodes   ┌─────────────┐
                 │   PENDING    ├──────────────────────────►│   RUNNING   │
                 │ (in queue)   │                           │ (on nodes)  │
                 └──────┬───────┘                           └──────┬──────┘
        scancel │                              ┌───────────────────┼───────────────┐
                ▼                              ▼                   ▼               ▼
          ┌───────────┐                 ┌────────────┐      ┌──────────┐    ┌──────────┐
          │ CANCELLED │                 │ COMPLETED  │      │  FAILED  │    │ TIMEOUT  │
          └───────────┘                 │ (exit 0)   │      │ (exit≠0) │    │ (walltime│
                                        └────────────┘      └──────────┘    │  hit)    │
                                                                            └──────────┘
   watch with: squeue (PENDING/RUNNING) → sacct (final states, post-mortem)

5. Command / Syntax / Code Breakdown¶

`module load gcc/12 openmpi/4.1`¶

Add the toolchain to PATH/LD_LIBRARY_PATH.

`module list`¶

Show currently loaded modules.

`module purge`¶

Unload everything; safe baseline.

`sbatch run.sbatch`¶

Submit a batch script — returns Submitted batch job 12345.

`squeue -u $USER`¶

Your jobs only.

`scancel 12345`¶

Cancel a job.

`srun --pty bash`¶

Drop into an interactive shell on the allocated node.

`salloc --nodes=2 --time=1:00:00`¶

Reserve resources, then run commands with srun.

`scontrol show job 12345`¶

Detailed status.

`sacct -j 12345 --format=JobID,State,Elapsed,MaxRSS`¶

Post-mortem accounting.

`#SBATCH --array=1-100`¶

Array job — 100 tasks indexed by $SLURM_ARRAY_TASK_ID.

`mpirun -n $SLURM_NTASKS ./solver`¶

Launch the MPI program with the right number of ranks.

6. Mandatory Practical Examples¶

Example 6.1 — First job: hello on cluster¶

hello.cpp:

#include <iostream>
#include <unistd.h>
int main(){
    char host[64]; gethostname(host,64);
    std::cout << "hello from " << host << "\n";
    return 0;
}

Build: g++ -O2 hello.cpp -o hello

Submit:

cat > run.sbatch <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=00:05:00
#SBATCH --output=hello.log
./hello
EOF
sbatch run.sbatch
squeue -u $USER
cat hello.log

Expected output: hello from node042

Example 6.2 — MPI job on cluster¶

cat > run_mpi.sbatch <<'EOF'
#!/bin/bash
#SBATCH --job-name=mpi_test
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --time=00:10:00
#SBATCH --output=mpi_%j.log
module load gcc/12 openmpi/4.1
mpirun -n $SLURM_NTASKS ./example01
EOF
sbatch run_mpi.sbatch

example01.cpp (Ch. 14 §6.1) prints rank from each of 8 ranks across 2 nodes.

Example 6.3 — Hybrid MPI + OpenMP¶

#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=2          # 2 MPI ranks per node
#SBATCH --cpus-per-task=8            # 8 OpenMP threads each
#SBATCH --time=04:00:00
module load gcc/12 openmpi/4.1
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
mpirun -n $SLURM_NTASKS --map-by ppr:2:node:pe=8 ./solver in.dat

Total cores used: 4 nodes × 2 ranks × 8 threads = 64 cores.

Example 6.4 — Array job for parameter sweep¶

#!/bin/bash
#SBATCH --job-name=Re_sweep
#SBATCH --array=0-9
#SBATCH --time=02:00:00
#SBATCH --ntasks=8
RE_LIST=(500 1000 2000 5000 10000 20000 50000 100000 200000 500000)
RE=${RE_LIST[$SLURM_ARRAY_TASK_ID]}
mkdir -p case_Re${RE} && cd case_Re${RE}
cp ../template/* .
sed -i -E "s/^Re=.*/Re=${RE}/" in.dat
mpirun -n $SLURM_NTASKS ../solver in.dat > run.log

Submit once: sbatch sweep.sbatch → 10 simulations run concurrently.

বাংলায়: Array job হলো cluster-এর parameter sweep-অস্ত্র: এক script, দশটা কপি, প্রতিটার আলাদা $SLURM_ARRAY_TASK_ID। bash array থেকে index দিয়ে Re-মান তোলা + sed দিয়ে input পালটানো — Ch 8-এর সব কৌশল এখানে একসাথে। পরীক্ষায় "10টা case একসাথে চালাও" মানেই --array।

Example 6.5 — Interactive debug¶

srun --pty --partition=testing --nodes=1 --ntasks-per-node=4 --time=1:00:00 bash
module load gcc/12 openmpi/4.1
mpirun -n 4 ./prog

Real-Life HPC/CFD Meaning. You almost always run a small interactive case before submitting a big batch.

Written-Exam Relevance. "Explain srun --pty bash" appears regularly.

Example 6.6 — Use module spider to find software¶

$ module spider openmpi
   openmpi/4.1   openmpi/5.0
   You will need to load all module(s) listed below before loading openmpi/4.1
       gcc/11   gcc/12

7. Real HPC/CFD Workflow¶

# (laptop)
git push origin main

# (cluster)
ssh hpc
cd $WORK/sim01
git pull
module purge
module load gcc/12 openmpi/4.1 cmake/3.27
cmake -B build -S .
cmake --build build -j

# Submit array sweep
sbatch sweep.sbatch

# Watch
watch -n 30 'squeue -u $USER'

# After completion
rsync -avzP results/ user@laptop:~/local/

8. Exercises and Solutions¶

The lecture has no dedicated exercise file for cluster usage; it is taught hands-on. Below are exam-grade self-made exercises.

Exercise 15-A — Write a SLURM script¶

Reserve 8 nodes × 16 tasks for 2 hours on partition standard, load gcc 12 + OpenMPI 4.1, run ./solver in.dat with MPI.

Solution:

#!/bin/bash
#SBATCH --job-name=cfd_run
#SBATCH --partition=standard
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=16
#SBATCH --time=02:00:00
#SBATCH --output=cfd_%j.log
module load gcc/12 openmpi/4.1
mpirun -n $SLURM_NTASKS ./solver in.dat

Marking scheme (8 marks): shebang / job-name / partition / nodes / ntasks-per-node / time / module load / mpirun.

Exercise 15-B — Inspect and cancel¶

squeue -u $USER
scancel 67890

Exercise 15-C — Module switching¶

Switch from gcc/12 + openmpi/4.1 to clang/15 + openmpi/5.0:

module switch gcc/12 clang/15
module switch openmpi/4.1 openmpi/5.0
module list

Exercise 15-D — Interactive node¶

1 node, 4 cores, 30 min on testing:

srun --partition=testing --nodes=1 --ntasks-per-node=4 --time=00:30:00 --pty bash

Common mistakes

Running heavy work on the login node.
Forgetting module load inside the script (the job does NOT inherit your shell's modules).
Asking for too many nodes for too long → never starts.
Using $HOME instead of $WORK for big I/O.
Mismatch between --ntasks and the mpirun -n value.

Harder version. Dependent jobs — run analysis only after the simulation finishes:

sbatch sim.sbatch              # → 12345
sbatch --dependency=afterok:12345 analyze.sbatch

9. Written Exam Focus¶

9.1 Short Answers¶

Q. Why are heavy jobs forbidden on login nodes? A. Login nodes are shared between many users; running compute-intensive code starves others. Schedulers exist to allocate compute nodes fairly.

Q. What is module load? A. A command that updates PATH, LD_LIBRARY_PATH, MANPATH etc. to make a particular software version available in the current shell.

Q. Difference between sbatch and srun. A. sbatch submits a non-interactive batch script and returns immediately. srun either starts an interactive task or launches a step inside an existing allocation.

Q. What is a partition? A. A named group of compute nodes with common properties (hardware, time limit). Target it with --partition=name.

Q. What is walltime? A. Maximum wall-clock time the job may run. SLURM kills it once exceeded.

9.2 Medium Answers¶

Q. (8 marks) Walk through the lifecycle of a CFD job on a SLURM cluster.

A. (1) SSH to login node; (2) git pull source; (3) module load gcc openmpi cmake; (4) build with cmake --build; (5) write run.sbatch with --nodes, --ntasks-per-node, --time; (6) sbatch run.sbatch; (7) squeue -u $USER while pending; (8) allocation → runs on compute nodes; (9) writes results to $WORK; (10) sacct reports stats; (11) rsync results to laptop.

Q. (5 marks) Compare salloc, srun, sbatch.

A. sbatch script queues a non-interactive script. salloc requests resources and gives you a shell from which you srun commands on the allocation. srun is the universal launcher inside an allocation; with --pty bash it gives an interactive shell on a compute node.

9.3 Long Answer (12 marks)¶

Q. Describe the architecture of a typical HPC cluster and how a CFD job traverses it.

A.

Introduction. An HPC cluster is a multi-tier Linux system tied together by a fast interconnect.

Main concept. Tiers: VPN gateway → login → compute (CPU/GPU/fat) → storage (Lustre/GPFS) → archive.

Step-by-step.

User SSH-es to a login node.
Edit and compile (modules pick compiler/MPI versions).
Submit with sbatch; SLURM queues it.
Scheduler allocates compute nodes when free.
Job script runs: loads modules, executes mpirun ./solver.
Solver writes to $WORK (parallel filesystem).
Post-processing (gnuplot, awk) on login/vis node.
Final results archived; code remains in Git.

Real CFD link. TU-BS Phoenix: 304 compute + 8 fat + 8 GPU nodes, OpenMPI, Lmod modules, SLURM.

Conclusion. Understanding cluster tiers and SLURM is essential to use the machine correctly and respectfully.

9.4 Output Prediction¶

$ sbatch run.sbatch
Submitted batch job 12345
$ squeue -u me
JOBID PARTITION NAME    USER      ST   TIME    NODES NODELIST
12345 standard cfd      me        R    0:01    4     node[001-004]

$ module list
Currently Loaded Modules:
  1) gcc/12   2) openmpi/4.1

9.5 Comparison¶

sbatch vs srun vs salloc — see 9.2. $HOME vs $WORK vs $ARCHIVE — small persistent vs big fast purged vs cold tape.

	Login	Compute
SSH-able	yes	only via job
Heavy use	no	yes
Lifetime	always on	per-job
Network	external	internal

9.6 Templates¶

Job-script template — see §4.6. Module template: module purge; module load <toolchain>. Cluster workflow: "ssh → edit → module → build → sbatch → squeue → results."

9.7 Marking Scheme — "Write SBATCH for MPI job" (10 marks)¶

1 shebang.
1 job-name + output.
1 partition.
1 nodes/ntasks.
1 cpus-per-task.
1 time.
1 module load.
1 export OMP_NUM_THREADS.
1 mpirun -n $SLURM_NTASKS.
1 cd $SLURM_SUBMIT_DIR or absolute paths.

10. Very Hard Questions¶

Beginner

Where does a job actually run? → On compute nodes.
List available software? → module avail.
Submit a job? → sbatch script.sh.
See the queue? → squeue.
Cancel a job? → scancel ID.

Intermediate

Reserve 1 node interactively. → srun --pty --nodes=1 bash.
Where to run heavy I/O? → $WORK.
Set OpenMP thread count from SLURM. → export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK.
Show details of job 99. → scontrol show job 99.
Module search. → module spider <name>.

Hard

Make jobs depend on previous. → sbatch --dependency=afterok:<id>.
ntasks vs cpus-per-task. → MPI ranks vs OpenMP threads per rank.
Pin processes to cores. → --cpu-bind=cores / OMP_PROC_BIND=true.
Why module purge first? → reproducible environment.
Partition info. → sinfo shows names + limits.

Very Hard

Why prefer sbatch over nohup? → scheduler accounting + isolation + restartability.
How does Lustre stripe files? → split across OSTs for parallel I/O.
Why is checkpointing critical for long jobs? → walltime-kill recovery.

Deep Integration

Git + module + SLURM integration → §7.
Why record module list? → reproducibility for the paper.

Coding/Command

srun for 4 ranks, 1 hour, partition gpu: srun --pty --partition=gpu --nodes=1 --ntasks=4 --gres=gpu:1 --time=1:00:00 bash
Bash one-liner to wait for your jobs: until [ -z "$(squeue -h -u $USER)" ]; do sleep 60; done

Debugging

"Permission denied" running job binary. → chmod +x or wrong path.
Job pending forever. → too many nodes/walltime; reduce or change partition.

Long Written

(250 words) Compare interactive salloc and batch sbatch workflows for CFD development.

11. Debugging and Mistake Analysis¶

Mistake	Why wrong	Correct	Explanation
Running on login node	shared resource	submit via SLURM	etiquette
`mpirun -n 16` without `--ntasks=16`	mismatch	use `$SLURM_NTASKS`	sync sbatch and mpirun
Forgot module load in script	binary not found	load explicitly	jobs get a cleansed env
Wrong partition	rejected	`sinfo` lists	check
`$HOME` for big I/O	quota / slow	`$WORK`	tier
Walltime too short	killed (TIMEOUT)	estimate + 25% margin	conservative
cpus-per-task ≠ OMP_NUM_THREADS	oversubscription	export from `$SLURM_CPUS_PER_TASK`	consistent
Forgot `cd $SLURM_SUBMIT_DIR`	runs in `$HOME`	always cd	known cwd

বাংলায়: Job-script debug-এর প্রথম তিন প্রশ্ন: (১) module কি script-এর ভেতরে load করা আছে? (২) --ntasks আর mpirun -n কি মিলছে? (৩) output ফাইলটা কোথায় লেখা হচ্ছে — $WORK নাকি ভুল করে $HOME? এই তিনটা মিলিয়ে নিলে ৮০% সমস্যা শেষ।

12. Mini Project for Mastery¶

Goal: Run a 10-case Reynolds sweep on the cluster.

# locally
ssh hpc
mkdir -p $WORK/sim_sweep && cd $WORK/sim_sweep
git clone git@gitlab:cfd/solver.git
cd solver
module purge && module load gcc/12 openmpi/4.1 cmake/3.27
cmake -B build && cmake --build build -j
cd ..

cat > sweep.sbatch <<'EOF'
#!/bin/bash
#SBATCH --job-name=Re_sweep
#SBATCH --array=0-9
#SBATCH --partition=standard
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --time=01:00:00
#SBATCH --output=sweep_%A_%a.log
module purge
module load gcc/12 openmpi/4.1
RE=(500 1000 2000 5000 10000 20000 50000 100000 200000 500000)
R=${RE[$SLURM_ARRAY_TASK_ID]}
DIR=$WORK/sim_sweep/case_Re${R}
mkdir -p $DIR && cd $DIR
cp $WORK/sim_sweep/template/* .
sed -i -E "s/^Re=.*/Re=${R}/" in.dat
mpirun -n $SLURM_NTASKS $WORK/sim_sweep/solver/build/solver in.dat
EOF

sbatch sweep.sbatch
squeue -u $USER

After all jobs finish, post-process with awk + gnuplot (Ch. 9 §7).

Connection to exam: module, sbatch array, environment vars, MPI, and the parameter-sweep idiom.

13. Final Chapter Cheat Sheet¶

Item	Memorise
SSH	`ssh user@cluster`
Module	`module purge && module load gcc/12 openmpi/4.1`
Submit	`sbatch run.sbatch`
Watch	`squeue -u $USER`
Cancel	`scancel ID`
Interactive	`srun --pty bash`
Allocate	`salloc --nodes=N --time=T`
Account	`sacct -j ID`
Partitions	`sinfo`
Storage	`$HOME` slow / `$WORK` fast / `$ARCHIVE` tape
Core-hours	nodes × cores/node × hours
Walltime	measured pilot × 1.25
Vars	`$SLURM_NTASKS`, `$SLURM_CPUS_PER_TASK`, `$SLURM_NODELIST`, `$SLURM_JOB_ID`, `$SLURM_SUBMIT_DIR`
Job header	`#SBATCH --nodes/--ntasks-per-node/--cpus-per-task/--time/--partition/--output`
Hybrid run	`OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; mpirun -n $SLURM_NTASKS ./prog`
States	PENDING → RUNNING → COMPLETED/FAILED/TIMEOUT/CANCELLED
Trap	running on login node
Top phrase	"Login = entry; SLURM allocates compute; modules pick versions; $WORK for I/O."

14. Mock Exam — Four Levels¶

Level 1 — Basic (definitions & syntax)¶

Q1. Give the commands to: submit job.sbatch, watch your jobs, cancel job 4242.

Solution: sbatch job.sbatch; squeue -u $USER; scancel 4242.

Q2. What do --nodes, --ntasks-per-node, and --time request?

Solution: Number of compute nodes; MPI tasks per node; maximum walltime.

Q3. Which command makes gcc 12 available in your shell?

Solution: module load gcc/12.

Q4. Where should a CFD job write its large output files?

Solution: $WORK (the fast parallel filesystem), not $HOME.

Q5. What does Submitted batch job 12345 give you?

Solution: The job ID — used with squeue/scancel/sacct/scontrol and %j in output names.

Level 2 — Intuitive (predict / explain why)¶

Q1. Your script works interactively but the batch job dies with "mpirun: command not found". Why?

Solution: Batch jobs start with a cleansed environment — your interactive module load doesn't carry over. Load modules INSIDE the script.

Q2. Two identical jobs: A requests --time=01:00:00, B requests --time=48:00:00. Which likely starts first and why?

Solution: A — backfill scheduling slots short jobs into gaps before big reservations; honest short walltimes reduce queue time.

Q3. squeue shows your job in state PD with reason "Resources". Meaning?

Solution: PENDING — waiting for the requested nodes/cores to become free; nothing is wrong.

Q4. A job with --ntasks=8 runs mpirun -n 16 ./solver. Predict the consequence.

Solution: 16 ranks crammed onto 8 allocated slots → oversubscription: 2 ranks per core, ~2× slowdown (or an MPI error, depending on configuration). Always use $SLURM_NTASKS.

Q5. Why does sacct -j ID --format=...,MaxRSS matter after a job FAILED?

Solution: MaxRSS reveals whether the job died from memory exhaustion (OOM) — the most common silent killer; you then request more memory or fewer ranks per node.

Level 3 — Hard (exam level)¶

Q1. (10 marks) Write a complete sbatch script: job name wing_les, partition standard, 4 nodes × 32 MPI ranks, 2 OpenMP threads per rank, 12 h walltime, log to wing_%j.log, modules gcc/12 + openmpi/4.1, run ./les in.cfg.

Solution:

#!/bin/bash
#SBATCH --job-name=wing_les
#SBATCH --partition=standard
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=2
#SBATCH --time=12:00:00
#SBATCH --output=wing_%j.log
module purge
module load gcc/12 openmpi/4.1
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
cd $SLURM_SUBMIT_DIR
mpirun -n $SLURM_NTASKS ./les in.cfg

বাংলা ইঙ্গিত: "32 MPI ranks per node + 2 threads" → ntasks-per-node=32, cpus-per-task=2; thread-সংখ্যা সবসময় $SLURM_CPUS_PER_TASK থেকে export — হাতে 2 লিখলে আধা নম্বর।

Q2. (8 marks) Compute the core-hours: 4 nodes × 48 cores, 6 h walltime, but the job finishes in 4.5 h. What is billed (typical policy), and what did the run actually consume?

Solution: Consumed: $4 \times 48 \times 4.5 = 864$ core-h. Typical billing charges actual elapsed time, not requested: 864 core-h (the 6 h request only affects scheduling). If a site bills by request, it would be $4\times48\times6 = 1152$ core-h — state the assumption. বাংলা ইঙ্গিত: সূত্র এক লাইনে: nodes × cores/node × ঘণ্টা; request বনাম actual কোনটা ধরছ — সেটা লিখে দিলে পরীক্ষক খুশি।

Q3. (8 marks) Budget a strong-scaling study: runs on 1, 2, 4, 8, 16, 32, 64 nodes (48 cores each), each run capped at 1 h walltime. Upper bound of core-hours for the whole study?

Solution: $\sum (1+2+4+8+16+32+64) = 127$ node-hours → $127 \times 48 = 6096$ core-hours upper bound. (Real cost is lower: bigger runs finish faster than the 1 h cap — that's the point of the study.) বাংলা ইঙ্গিত: doubling-series-এর যোগফল $2^{k+1}-1$ — এখানে 127; তারপর শুধু × cores।

Q4. (10 marks) This jobscript fails instantly with exit code 127 and an empty log except "solver: command not found". Find ALL problems:

#!/bin/bash
#SBATCH --nodes=2 --ntasks-per-node=16
#SBATCH --time=24
mpirun -n 32 solver in.dat

Solution: (1) No module load → MPI/compiler runtime missing; (2) solver without ./ or absolute path → not on PATH (the "command not found", exit 127); (3) --time=24 means 24 minutes, not hours — write --time=24:00:00; (4) hard-coded -n 32 instead of $SLURM_NTASKS; (5) (style) two directives on one line are legal but error-prone — one per line. Corrected script loads modules, uses ./solver, proper time format. বাংলা ইঙ্গিত: exit 127-এর মানেই "command not found" — PATH-জাতীয় রোগ; আর SLURM-এর time format ফাঁদ: একা সংখ্যা = মিনিট।

Q5. (10 marks) Explain #SBATCH --array=0-99%10 and write the line inside the script that picks the parameter for each task from a bash array P=( ... ).

Solution: An array job of 100 tasks (indices 0–99), with at most 10 running simultaneously (%10 throttle — polite to the cluster). Selection line: VAL=${P[$SLURM_ARRAY_TASK_ID]}. বাংলা ইঙ্গিত: %10-এর মানে জানা থাকলে এটা free marks — throttle; না জানলে অনুমান করা কঠিন।

Level 4 — Beyond the lecture (transfer + coding)¶

Q1. Design a checkpoint-restart chain for a 100 h simulation on a cluster with a 24 h walltime limit. Give the sbatch mechanics.

Solution: The solver must write checkpoints (e.g. every 30 min). Submit a chain in which each job restarts from the last checkpoint and resubmits its successor:

# inside run.sbatch, at the end:
if [ ! -f converged.flag ]; then
    sbatch --dependency=afterany:$SLURM_JOB_ID run.sbatch
fi
mpirun -n $SLURM_NTASKS ./solver --restart latest.chk in.dat

(Or pre-submit 5 jobs with --dependency=afterany.) afterany (not afterok) ensures the chain continues even after a TIMEOUT kill, which is the expected end of each segment. বাংলা ইঙ্গিত: afterok বনাম afterany-এর পার্থক্যই প্রশ্নের আত্মা — TIMEOUT মানে exit≠0, তাই afterok-এ chain মরে যেত।

Q2. Combine Ch 7 + Ch 15: after an array sweep, write ONE pipeline that scans all sweep_*_*.log files, extracts the final residual (lines like FINAL residual = 1.2e-6), and prints the worst (largest) value with its filename.

Solution:

grep -H "FINAL residual" sweep_*_*.log \
  | sed -E 's/:.*= */ /' \
  | sort -k2 -g \
  | tail -1

-H forces filenames; sed reduces to "file value"; sort -g handles scientific notation (-n would not!); tail takes the largest. বাংলা ইঙ্গিত: 1.2e-6-জাতীয় সংখ্যায় sort -n ব্যর্থ — general-numeric -g লাগবে; এই এক flag-ই প্রশ্নটার আসল দাঁত।

Q3. Your MPI job runs at 50% of expected speed on 2 nodes but full speed on 1 node. Hypothesise the cause and give the SLURM/mpirun flags to test the hypothesis.

Solution: Hypothesis: inter-node communication bottleneck — ranks placed across nodes communicate over the network instead of shared memory, or the job didn't land on the fast interconnect. Tests: check placement srun --ntasks=32 hostname | sort | uniq -c; force compact placement mpirun --map-by node --bind-to core --report-bindings; compare 32 ranks on 1 node vs 16+16 on 2 nodes. If 16+16 is slow, the halo traffic crosses the slow path → check partition/interconnect (e.g. request the IB-connected partition). বাংলা ইঙ্গিত: "1 node ভালো, 2 node খারাপ" শুনলেই সন্দেহ network-এ — placement আর binding প্রিন্ট করানোই প্রথম পরীক্ষা।

Q4. Write a small bash monitor (Ch 8 transfer) that emails (or echoes) a summary when YOUR last submitted job finishes: job ID captured from sbatch, polls every 60 s, then prints state + elapsed + MaxRSS from sacct.

Solution:

#!/bin/bash
set -euo pipefail
JID=$(sbatch "$1" | awk '{print $4}')
echo "submitted $JID"
while squeue -h -j "$JID" 2>/dev/null | grep -q .; do
    sleep 60
done
sacct -j "$JID" --format=JobID,State,Elapsed,MaxRSS | tee summary.txt

sbatch output's 4th word is the ID; the loop ends when squeue no longer lists the job; sacct delivers the post-mortem. বাংলা ইঙ্গিত: sbatch ... | awk '{print $4}' — job ID ধরার idiom; আর "job শেষ" মানে squeue-তে আর নেই — এই দুটো টুকরাই script-টার কঙ্কাল।

End of Chapter 15. ap.

Solution: Request $\approx 1.25 \times 4$ h $= 5$ h → #SBATCH --time=05:00:00. Too short → TIMEOUT kills the job and all uncheckpointed progress is gone; too long → harder to backfill (longer queue wait) and worse scheduling priority, though nothing is lost. For the 10-day campaign: the walltime cap (48 h) is shorter than the simulation, so checkpointing is mandatory — write a restart file e.g. every 2 h, submit a chain of jobs each restarting from the last checkpoint, automated with sbatch --dependency=afterok:<previous_jobid> next.sbatch (or afterany if a TIMEOUT-killed job should still trigger the next restart).

বাংলা ইঙ্গিত: 10 দিনের simulation আর 48 ঘণ্টার cap দেখলেই বুঝে নাও প্রশ্নটা আসলে checkpoint + dependency chain-এর — walltime বাড়িয়ে চাওয়ার কোনো উপায়ই নেই।

Level 4 — Beyond the lecture (transfer questions)¶

Q4.1 Write (a) an array-job script that runs a mesh-convergence study over 6 meshes mesh_050k.cgns … mesh_1600k.cgns, and (b) the one-line submission of an analysis job that starts only after all array tasks succeed.

Solution:

(a) conv.sbatch:

#!/bin/bash
#SBATCH --job-name=mesh_conv
#SBATCH --array=0-5
#SBATCH --partition=standard
#SBATCH --nodes=1
#SBATCH --ntasks=48
#SBATCH --time=06:00:00
#SBATCH --output=conv_%A_%a.log
module purge
module load gcc/12 openmpi/4.1
MESHES=(mesh_050k mesh_100k mesh_200k mesh_400k mesh_800k mesh_1600k)
M=${MESHES[$SLURM_ARRAY_TASK_ID]}
DIR=$WORK/conv/$M
mkdir -p $DIR && cd $DIR
cp $WORK/conv/template/* .
sed -i -E "s/^mesh=.*/mesh=${M}.cgns/" in.dat
mpirun -n $SLURM_NTASKS $WORK/conv/solver in.dat

(b) Submit and chain (%A is the array master ID):

JID=$(sbatch --parsable conv.sbatch)
sbatch --dependency=afterok:${JID} analyze.sbatch

afterok on an array job ID waits for all array tasks to end successfully; if any mesh fails, the analysis never starts (use afterany to run it regardless).

বাংলা ইঙ্গিত: --parsable দিয়ে job ID টা variable-এ ধরাই script-চেইনের পেশাদার কৌশল — output থেকে হাতে নম্বর টোকার দরকার নেই।

Q4.2 Write a bash script submit_scaling.sh that generates and submits one sbatch script per node count $N \in \{1,2,4,8,16\}$ for a strong-scaling study (48 ranks per node), using a here-document.

Solution:

#!/bin/bash
set -e
for N in 1 2 4 8 16; do
    cat > scale_${N}.sbatch <<EOF
#!/bin/bash
#SBATCH --job-name=scale_${N}
#SBATCH --partition=standard
#SBATCH --nodes=${N}
#SBATCH --ntasks-per-node=48
#SBATCH --time=10:00:00
#SBATCH --output=scale_${N}_%j.log
module purge
module load gcc/12 openmpi/4.1
cd \$SLURM_SUBMIT_DIR
echo "nodes=${N} ranks=\$SLURM_NTASKS start=\$(date +%s)" >> timing.txt
mpirun -n \$SLURM_NTASKS ./solver in.dat
echo "nodes=${N} end=\$(date +%s)" >> timing.txt
EOF
    sbatch scale_${N}.sbatch
done

The trap: inside the unquoted here-document, ${N} must expand now (generation time) but $SLURM_NTASKS must survive into the script — hence the escaped \$SLURM_NTASKS and \$(date ...). Afterwards timing.txt provides $T_N$ for the speed-up table (and §4.10 predicts the total budget).

বাংলা ইঙ্গিত: Heredoc-এর ভেতরে কোন $ এখনই বসবে আর কোনটা job চলার সময় বসবে — এই escape-এর খেলাটাই প্রশ্নের আসল ফাঁদ।

Q4.3 You compile your solver on the login node with g++ -O3 -march=native and it dies on the compute node with Illegal instruction (core dumped). Explain the failure and give two correct workflows.

Solution: -march=native tells the compiler to emit instructions for the CPU doing the compiling — the login node. If the login node is a newer CPU generation (e.g. supports AVX-512) than the compute nodes, the binary contains instructions the compute node's CPU literally cannot execute → SIGILL, Illegal instruction. Fixes: (1) compile on a compute node: srun --partition=standard --nodes=1 --time=00:30:00 --pty bash, then build there (or make compilation the first step of the job script); (2) target the compute-node architecture explicitly from the login node, e.g. g++ -O3 -march=skylake-avx512 (whatever the compute nodes are), or use a portable baseline like -O3 -march=x86-64-v3. Verifying with gdb ./solver core would show the crash at the first illegal vector instruction — connecting Chapter 13 to cluster reality.

বাংলা ইঙ্গিত: "Illegal instruction" + cluster = প্রায় সবসময় -march=native-এর গল্প: যেখানে compile, সেখানকার CPU-র নির্দেশ; যেখানে run, সেখানে তা নেই।

Q4.4 A CFD case needs 100 M cells × 5 unknowns × 8 bytes ≈ 4 GB of field data, but with solver overhead the real footprint is ~480 GB RAM. Nodes have 192 GB and 48 cores. (a) Minimum number of nodes? (b) Write the resource part of the sbatch script. (c) Cost of a 24 h run on that minimum.

Solution:

(a) Memory, not cores, is binding: $\lceil 480 / 192 \rceil = \lceil 2.5 \rceil = 3$ nodes (576 GB total).

(b)

#SBATCH --nodes=3
#SBATCH --ntasks-per-node=48        # 144 ranks; per-rank memory 480/144 = 3.3 GB
#SBATCH --mem=0                     # claim all memory of each node
#SBATCH --time=24:00:00

(--mem=0 requests full node memory; alternatively --mem-per-cpu=4G since $48 \times 4 = 192$ GB.)

(c) $3 \times 48 \times 24 = 3456$ core-h.

বাংলা ইঙ্গিত: Node সংখ্যা সবসময় core থেকে আসে না — আগে দেখো কোন resource (RAM নাকি core) আগে ফুরায়, তারপর ceiling নাও।

End of Chapter 15.

Chapter 15: HPC Cluster Usage — modules, SLURM, jobs¶

1. Chapter Overview¶

2. Basics from Zero¶

3. Hard English Made Easy¶

4. Deep Theory Explanation¶

4.1 Cluster topology (slide 8)¶

4.2 Cluster nodes by type (slide 9)¶

4.3 Storage tiers (slide 11)¶

4.4 Modules¶

4.5 SLURM commands¶

4.6 Anatomy of a SLURM batch script¶

4.7 Interactive session¶

4.8 Choosing partition / time / size¶

4.9 Diagrams from V13¶

4.10 Compute-budget arithmetic — core-hours, queues, walltime¶

4.11 SLURM job-state lifecycle¶

5. Command / Syntax / Code Breakdown¶

module load gcc/12 openmpi/4.1¶

module list¶

module purge¶

sbatch run.sbatch¶

squeue -u $USER¶

scancel 12345¶

srun --pty bash¶

salloc --nodes=2 --time=1:00:00¶

scontrol show job 12345¶

sacct -j 12345 --format=JobID,State,Elapsed,MaxRSS¶

#SBATCH --array=1-100¶

mpirun -n $SLURM_NTASKS ./solver¶

6. Mandatory Practical Examples¶

Example 6.1 — First job: hello on cluster¶

Example 6.2 — MPI job on cluster¶

Example 6.3 — Hybrid MPI + OpenMP¶

Example 6.4 — Array job for parameter sweep¶

Example 6.5 — Interactive debug¶

Example 6.6 — Use module spider to find software¶

7. Real HPC/CFD Workflow¶

8. Exercises and Solutions¶

Exercise 15-A — Write a SLURM script¶

Exercise 15-B — Inspect and cancel¶

Exercise 15-C — Module switching¶

Exercise 15-D — Interactive node¶

9. Written Exam Focus¶

9.1 Short Answers¶

9.2 Medium Answers¶

9.3 Long Answer (12 marks)¶

9.4 Output Prediction¶

9.5 Comparison¶

9.6 Templates¶

9.7 Marking Scheme — "Write SBATCH for MPI job" (10 marks)¶

10. Very Hard Questions¶

11. Debugging and Mistake Analysis¶

12. Mini Project for Mastery¶

13. Final Chapter Cheat Sheet¶

14. Mock Exam — Four Levels¶

Level 1 — Basic (definitions & syntax)¶

Level 2 — Intuitive (predict / explain why)¶

Level 3 — Hard (exam level)¶

Level 4 — Beyond the lecture (transfer + coding)¶

Level 4 — Beyond the lecture (transfer questions)¶

`module load gcc/12 openmpi/4.1`¶

`module list`¶

`module purge`¶

`sbatch run.sbatch`¶

`squeue -u $USER`¶

`scancel 12345`¶

`srun --pty bash`¶

`salloc --nodes=2 --time=1:00:00`¶

`scontrol show job 12345`¶

`sacct -j 12345 --format=JobID,State,Elapsed,MaxRSS`¶

`#SBATCH --array=1-100`¶

`mpirun -n $SLURM_NTASKS ./solver`¶