Skip to content

Chapter 1: Course Introduction and HPC Overview

Course: Fundamentals of High-Performance Computing for CFD Simulations
Lecturer: Jun.-Prof. Dr.-Ing. Federica Ferraro (IFAS, Working Group: Reactive Flows in Aero Engines)
Source slides: L01_Introduction_and_Linux.pdf (slides 1 ~ 35), read.pdf slides 1 ~ 22

1. Chapter Overview

This first chapter is the conceptual foundation of the entire course. It tells you why an engineer who works on CFD (Computational Fluid Dynamics) must master Linux, scripting, compilers, and parallel computing.

What it is about:

  • Who teaches the course (IFAS — Institute of Jet Propulsion and Turbomachinery, Leibniz Universität Hannover).
  • The research goal: low-emission, energy-efficient aircraft propulsion, simulated with HPC.
  • Why HPC (High-Performance Computing) is essential for CFD.
  • The role of supercomputers (Top500, El Capitan, Frontier, JUWELS).
  • The organisation of the course (block course, dates, hands-on sessions).
  • A first encounter with Unix/Linux and the reason every engineering simulation environment uses Linux.

Why it matters in HPC and CFD:

  • Real CFD simulations (e.g. of a combustion chamber in a gas turbine) generate terabytes of data and need millions of CPU-core-hours. Without parallel computing on a cluster you simply cannot finish them.
  • Every HPC cluster on earth runs Linux. There is no Windows supercomputer in the Top500 list.
  • Mastering the workflow (Linux, SSH, shell scripting, compiling, debugging, plotting, version control) is a prerequisite for doing CFD research at all.

What the examiner may ask:

  • "Define HPC. Why is HPC necessary for CFD?"
  • "Name three reasons why Linux dominates supercomputing."
  • "What is the Top500 list?"
  • "Describe the difference between experiments and virtual experiments in turbomachinery research."
  • Short factual / definition questions (3–5 marks).

What you must master for top grade:

  • A clean two-line definition of HPC that you can write under exam pressure.
  • One concrete CFD example (gas turbine combustion chamber) you can describe in 4–5 sentences.
  • The four "supercomputer facts": parallelism, FLOPS unit, Top500, Linux dominance.
  • The four reasons Linux is preferred for HPC: stability, scriptability, free/open, modular.

2. Basics from Zero

Imagine you want to simulate the airflow inside a jet engine. The engine is so complicated that the equations describing the gas (Navier–Stokes) cannot be solved by hand. Instead, you split the engine into millions of tiny cells, write the equations for each cell, and solve all of them at the same time on a computer.

A normal laptop has 4–8 cores. With 8 cores the simulation might take 6 months. A modern supercomputer has millions of cores and can do the same simulation in a few hours. That ability — to run a single huge calculation across many cores at the same time — is what we call High-Performance Computing (HPC).

বাংলায়: HPC-র মূল ধারণা খুবই সহজ: একটা বিশাল হিসাবকে ছোট ছোট টুকরোয় ভেঙে হাজার হাজার core-এ একসাথে চালানো। Laptop-এ যে simulation ৬ মাস লাগত, supercomputer-এ তা কয়েক ঘণ্টায় শেষ হয়, কারণ কাজটা parallel-এ ভাগ হয়ে যায়। পরীক্ষায় দুই লাইনের definition চাওয়া হয় — "many computers working on one calculation" — এটা মুখস্থ রাখুন।

The supercomputer is not magic: it is just a lot of normal computers ("nodes") connected by a fast network. To use it you must connect remotely (SSH), copy your input files (scp/rsync), compile your code (g++/icc/mpicc), run it as a batch job (SLURM), wait, and then post-process the results (gnuplot, ParaView).

Every step of that workflow happens inside a Linux terminal. That is why this entire course teaches Linux first — it is the language of HPC.

বাংলায়: Cluster-এ কাজ করার পুরো রাস্তাটা মনে রাখুন: SSH দিয়ে login, scp দিয়ে file পাঠানো, compile করা, SLURM-এ job জমা দেওয়া, শেষে gnuplot দিয়ে ফলাফল আঁকা — প্রতিটা ধাপই Linux terminal-এ ঘটে। সেজন্যই Linux-কে HPC-র ভাষা বলা হয়; terminal না জানলে supercomputer ব্যবহার করাই অসম্ভব।

Real-life analogy. A normal computer is one chef cooking one meal. A supercomputer is a stadium full of 10 000 chefs all cooking together — but only if there is a head-cook (you, the user) who knows how to give commands so they don't all collide. Linux + the shell is that command system.

What happens if you misunderstand? You write a serial code that runs only on one core, queue it on a 1024-core node, the cluster's accounting bills you for 1024 cores × 24 h = 24 576 core-hours wasted. Real money, real CO₂.

3. Hard English Made Easy

Hard Term Simple English বাংলা Example
High-Performance Computing (HPC) Using many computers together to solve big problems fast অনেক কম্পিউটার একসাথে কাজ করিয়ে বড় সমস্যা দ্রুত সমাধান করা Frontier solves climate models in hours
Computational Fluid Dynamics (CFD) Solving fluid-flow equations on a computer কম্পিউটারে তরল পদার্থের প্রবাহ সমাধান করা Simulating air around a wing
Parallel computing Doing many calculations at the same time একই সময়ে অনেকগুলি হিসাব করা 1000 cores compute 1000 cells simultaneously
Supercomputer A very large parallel computer বিশাল সমান্তরাল কম্পিউটার El Capitan, Frontier, JUWELS
Top500 A list of the 500 fastest supercomputers বিশ্বের ৫০০ দ্রুততম সুপারকম্পিউটারের তালিকা top500.org
FLOPS Floating-point operations per second (a measure of speed) প্রতি সেকেন্ডে দশমিক হিসাব সংখ্যা 1 ExaFLOPS = \(10^{18}\) ops/s
Cluster A group of connected computers একসাথে যুক্ত অনেক কম্পিউটার A cluster has many nodes
Node One physical computer inside a cluster ক্লাস্টারের একটি কম্পিউটার A node has 64 cores
Core One processing unit inside a CPU প্রসেসরের একটি ছোট অংশ 16 cores per CPU
Virtual experiment A simulation that replaces a physical test কম্পিউটারে চালানো ভার্চুয়াল পরীক্ষা Simulating a combustor without building it
SAF Sustainable Aviation Fuel (drop-in green fuel) পরিবেশ বান্ধব বিমান জ্বালানি Used to lower CO₂ emissions
Sustainable aviation Aviation with low CO₂ impact পরিবেশসম্মত বিমান চলাচল Cluster of Excellence SE²A

4. Deep Theory Explanation

4.1 What is HPC?

Formal definition. HPC is the practice of aggregating computing power so that the resulting performance is much higher than a typical desktop, in order to solve problems that require very large datasets, very many calculations, or very small time-steps.

Intuition. "Many computers, one calculation." Instead of 1 CPU running for 1 year, you let 8 760 CPUs run for 1 hour.

How it works internally. A supercomputer is built from compute nodes (each like a powerful workstation) connected by a high-speed interconnect (e.g. InfiniBand, Slingshot-11). A job scheduler (SLURM) gives the user a slice of the machine. The user's program is split — through MPI (between nodes) and OpenMP (inside one node) — across thousands of cores. Specialised hardware (GPUs, AMD MI300A, NVIDIA H100) accelerates each step.

Why useful. It is the only way to do realistic CFD: detailed turbulence in a gas turbine combustor needs \(\approx 10^{9}\) grid cells \(\times\, 10^{5}\) time steps. No laptop can finish that.

Assumptions / limitations. HPC requires that the algorithm can be parallelised. Some algorithms cannot (Amdahl's law). Communication between nodes can become the bottleneck.

Common mistakes. Confusing "parallel" with "fast". Adding more cores does not always mean faster — overheads grow.

Written-exam relevance. Definition + 1 example + 1 limitation. Standard 4-mark answer.

বাংলায়: HPC মানে অনেক node-কে দ্রুত interconnect দিয়ে জুড়ে একটাই প্রোগ্রাম MPI (node-গুলোর মাঝে) আর OpenMP (একটা node-এর ভেতরে) দিয়ে হাজারো core-এ ভাগ করে চালানো। সাবধান: parallel মানেই দ্রুত নয় — Amdahl's law অনুযায়ী code-এর serial অংশই speed-up-এর সীমা ঠিক করে দেয়। পরীক্ষায় definition + একটা example + একটা limitation লিখলেই পুরো নম্বর।

4.2 Why CFD needs HPC

Navier–Stokes equations have no analytical solution for turbulent flow. Discretisation (Finite-Volume Method) on a fine 3-D mesh produces a sparse linear system with millions of unknowns per time step. A Direct Numerical Simulation (DNS) of a small combustor needs \(\approx 10^{15}\) floating-point operations.

At a modern node speed of ≈ 1 TFLOPS that is \(10^{15} / 10^{12}\) s \(\approx 10^{3}\) s \(\approx\) 17 minutes — but only if all cores are fully busy and communication is free. Real life: factor 10–100 slower. Hence: use thousands of nodes.

বাংলায়: Turbulent flow-এর জন্য Navier–Stokes equation-এর কোনো analytical সমাধান নেই, তাই mesh-এর কোটি কোটি cell-এ numerically হিসাব করতে হয়। একটা ছোট combustor-এর DNS-তেই প্রায় \(10^{15}\) operation লাগে — কোনো একক workstation-এ এটা অসম্ভব। এই সংখ্যাটাই "Why does CFD need HPC?" প্রশ্নের উত্তরের মেরুদণ্ড।

4.3 The Top500 list

A semi-annual ranking of the world's 500 fastest publicly-known computers, measured by the HPL (LINPACK) benchmark in FLOPS. As of mid-2024:

  • 1: El Capitan (LLNL, USA) — HPE Cray EX255a, AMD 4th-Gen EPYC + AMD MI300A, ~1.7 ExaFLOPS Rmax.

  • 2: Frontier (Oak Ridge, USA).

  • 3: Aurora (Argonne, USA).

  • All ≥ #1 run Linux.

বাংলায়: Top500 হলো বছরে দু'বার (জুন ও নভেম্বর) প্রকাশিত দ্রুততম ৫০০ supercomputer-এর তালিকা, যেখানে গতি মাপা হয় LINPACK (HPL) benchmark-এর \(R_{max}\) দিয়ে। দুটো তথ্য short question-এ বারবার আসে: শীর্ষ মেশিন El Capitan প্রায় 1.7 EFLOPS, আর তালিকার ১০০% মেশিনই Linux চালায়।

4.4 Linux dominance in HPC

Reason Why it matters
Open-source Vendors can patch the kernel for new hardware.
Stable Cluster nodes run unattended for months.
Modular You install only what you need on each node.
Scriptable Job-control / automation via shell scripts.
POSIX-compliant Portable software across distributions.

বাংলায়: Linux কেন HPC-তে একচেটিয়া — চারটে কারণ মুখস্থ করুন: open-source (vendor কার্নেল বদলাতে পারে), stable (মাসের পর মাস reboot ছাড়া চলে), modular (node-এ শুধু দরকারি অংশ থাকে), scriptable (সব কাজ shell দিয়ে automate করা যায়)। "Name three reasons" ধরনের প্রশ্নে এই তালিকা থেকেই তিনটে লিখবেন।

4.5 The course structure

Block course May ~ mid-July, Tue & Thu 12:00–15:00, lecture + hands-on. Topics in order: Linux → SSH/bashrc → Vim → Regex → text tools → shell scripting → gnuplot → Git → C++ → build process → debugging → parallelisation → cluster usage. Exam at end of July.

4.6 Diagram interpretations from L01

  • Slide "Experiments vs Virtual experiments": two photographs side-by-side (a flame in a test rig, a coloured CFD field). Left = costly, ground-truth, limited points. Right = simulation, complementary to experiments, cheap to repeat. In a written answer: describe the trade-off — experiments give truth but few quantities; simulations give all quantities but need validation by experiments.
  • "Top500 (June 2023)" pie chart: market shares of architectures (AMD, Intel, IBM POWER, ARM). Take-away to write: "Modern supercomputers are predominantly heterogeneous (CPU + GPU/accelerator)."
  • "Course structure" calendar: lectures and hands-on alternate, exam in last week of July.

4.7 Theoretical peak performance \(R_{peak}\) — the formula you must compute with

Every HPC machine has a theoretical peak performance: the number of floating-point operations per second it would deliver if every arithmetic unit were busy in every single clock cycle.

\[R_{peak} = N_{\text{nodes}} \times N_{\text{sockets/node}} \times N_{\text{cores/socket}} \times f_{\text{clock}} \times \frac{\text{FLOP}}{\text{cycle}}\]

The last factor is fixed by the vector (SIMD) hardware of one core:

\[\frac{\text{FLOP}}{\text{cycle}} = \underbrace{N_{\text{SIMD lanes}}}_{\text{vector width}} \times \underbrace{2}_{\text{FMA: } a\cdot b + c} \times N_{\text{FMA units}}\]
Vector ISA Register width DP lanes FMA units DP FLOP/cycle
SSE2 128 bit 2 1 4
AVX2 + FMA 256 bit 4 2 16
AVX-512 (2 FMA units) 512 bit 8 2 32

Fully worked example (write it step-by-step exactly like this in the exam). A node has 2 sockets × 24 cores at \(f_{\text{clock}} = 2.4\) GHz, with AVX-512 and two FMA units per core.

  1. FLOP per cycle per core (double precision): \(8 \times 2 \times 2 = 32\).
  2. Cores per node: \(2 \times 24 = 48\).
  3. Cycles per second of the whole node: \(48 \times 2.4 \times 10^{9} = 1.152 \times 10^{11}\).
  4. Peak per node (DP): \(1.152 \times 10^{11} \times 32 = 3.686 \times 10^{12} \approx 3.69\) TFLOP/s.
  5. Single precision has twice the lanes (64 FLOP/cycle): \(2 \times 3.69 \approx 7.37\) TFLOP/s (SP) per node.
  6. Scale to a 100-node cluster: \(100 \times 3.69 = 368.6\) TFLOP/s \(\approx 0.37\) PFLOP/s (DP).

Where these cores physically live:

┌─────────────────────────── CLUSTER ────────────────────────────┐
│ ≈ 100 nodes, joined by a fast interconnect (InfiniBand)        │
│                                                                │
│  ┌──────────────────────── RACK ──────────────────────────┐    │
│  │ ≈ 20 nodes per cabinet (5 racks in this example)       │    │
│  │                                                        │    │
│  │   ┌────────────────────── NODE ────────────────────┐   │    │
│  │   │ one "computer": 2 sockets + RAM + network card │   │    │
│  │   │                                                │   │    │
│  │   │   ┌───────────── SOCKET (CPU) ─────────────┐   │   │    │
│  │   │   │ 24 cores sharing one L3 cache          │   │   │    │
│  │   │   │                                        │   │   │    │
│  │   │   │   ┌──────────── CORE ──────────────┐   │   │   │    │
│  │   │   │   │ 1 instruction stream           │   │   │   │    │
│  │   │   │   │ SIMD: 8 doubles / instruction  │   │   │   │    │
│  │   │   │   │ 2 FMA units ──► 32 FLOP/cycle  │   │   │   │    │
│  │   │   │   └────────────────────────────────┘   │   │   │    │
│  │   │   └────────────────────────────────────────┘   │   │    │
│  │   └────────────────────────────────────────────────┘   │    │
│  └────────────────────────────────────────────────────────┘    │
└────────────────────────────────────────────────────────────────┘
Counts here: 100 nodes × 2 sockets × 24 cores = 4 800 cores total

Important caveat: \(R_{peak}\) is never reached. HPL/LINPACK sustains 60–80% of it (\(R_{max}\)); real CFD codes sustain 1–10% (see machine balance below).

বাংলায়: \(R_{peak}\) হলো কাগজে-কলমে সর্বোচ্চ গতি — প্রতিটা factor আলাদা লাইনে লিখে তারপর গুণ করুন: node × socket × core × clock × FLOP/cycle। FLOP/cycle আসে vector width (AVX-512-এ ৮টা double) গুণ ২ (FMA-তে multiply আর add একসাথে) গুণ FMA unit সংখ্যা থেকে। পরীক্ষায় ধাপগুলো আলাদা দেখালে আংশিক নম্বরও পাওয়া যায়, তাই কখনো এক লাইনে শুধু চূড়ান্ত উত্তর লিখবেন না।

4.8 FLOPS prefixes — memorise the whole ladder

Unit Value In words Typical machine
MFLOPS \(10^{6}\) FLOP/s million 1980s workstation
GFLOPS \(10^{9}\) FLOP/s billion one laptop core
TFLOPS \(10^{12}\) FLOP/s trillion one modern node / one GPU
PFLOPS \(10^{15}\) FLOP/s quadrillion mid-size cluster
EFLOPS \(10^{18}\) FLOP/s quintillion El Capitan, Frontier (exascale)

Each step is a factor of 1000. "Exascale" means \(R_{max} \geq 1\) EFLOPS on HPL.

4.9 Machine balance (bytes per FLOP) — why CFD never reaches peak

The machine balance \(B_m\) compares how fast memory can feed the cores with how fast the cores can compute:

\[B_m = \frac{\text{memory bandwidth [byte/s]}}{R_{peak}\ [\text{FLOP/s}]} \quad \left[\frac{\text{byte}}{\text{FLOP}}\right]\]

The code balance \(B_c\) is the same ratio for your loop: bytes that must be moved per FLOP executed. The achievable fraction of peak is approximately \(\min(1, B_m / B_c)\).

Fully worked example. Take the node from §4.7: \(R_{peak} = 3.6864\) TFLOP/s (DP) and memory bandwidth 200 GB/s.

  1. \(B_m = \dfrac{200 \times 10^{9}}{3.6864 \times 10^{12}} \approx 0.054\) byte/FLOP.
  2. A triad loop a(i) = b(i) + s*c(i) does 2 FLOP per iteration and moves 24 bytes (load b, load c, store a, 8 bytes each): \(B_c = 24/2 = 12\) byte/FLOP.
  3. Achievable fraction of peak: \(0.054 / 12 \approx 0.45\%\) — about 17 GFLOP/s out of 3 686 GFLOP/s.

This is the quantitative reason why CFD (a bandwidth-hungry stencil/sparse workload) sustains only a few percent of \(R_{peak}\). Cache reuse improves \(B_c\) and pushes the fraction up toward the typical 1–10%.

বাংলায়: Machine balance (\(B_m\)) বলে memory প্রতি FLOP-এ কত byte জোগান দিতে পারে, আর code balance (\(B_c\)) বলে আপনার loop-এর প্রতি FLOP-এ কত byte দরকার। আজকের মেশিনে \(B_m \approx 0.05\) byte/FLOP, কিন্তু সাধারণ CFD loop-এর দরকার প্রায় 10 byte/FLOP — তাই CFD প্রায় সবসময় memory-bound, পিক-এর মাত্র কয়েক শতাংশ পায়। এটাই ব্যাখ্যা করে কেন \(R_{max} \ll R_{peak}\)।

4.10 The memory hierarchy — sizes and latencies (order-of-magnitude!)

All latency numbers below are order-of-magnitude values for a ~3 GHz CPU; exact values vary by architecture. The ordering is what the exam checks.

Level Typical size Latency (cycles) Latency (≈ ns)
Registers ~ kB ~1 ~0.3
L1 cache 32–64 KiB per core ~4 ~1.3
L2 cache 0.5–2 MiB per core ~12 ~4
L3 cache 32–256 MiB per socket ~40 ~13
RAM (DDR/HBM) 0.1–2 TiB per node ~200 ~70
Disk / parallel FS PB per cluster \(10^{5}\)–\(10^{7}\) µs–ms 
                        ▲ smaller, faster, more expensive
                        │
                  ┌───────────┐
                  │ Registers │              ~1 cycle,   ~kB
                ┌─┴───────────┴─┐
                │   L1 cache    │            ~4 cycles,  32–64 KiB/core
              ┌─┴───────────────┴─┐
              │     L2 cache      │          ~12 cycles, 0.5–2 MiB/core
            ┌─┴───────────────────┴─┐
            │       L3 cache        │        ~40 cycles, 32–256 MiB/socket
          ┌─┴───────────────────────┴─┐
          │       RAM (DDR/HBM)       │      ~200 cycles, 0.1–2 TiB/node
        ┌─┴───────────────────────────┴─┐
        │ Disk / parallel FS (Lustre)   │    ~10^5–10^7 cycles, PB/cluster
        └───────────────────────────────┘
                        │
                        ▼ bigger, slower, cheaper

A RAM access costs ~200 cycles during which an AVX-512 core could have done ~6 400 FLOP. That is why cache-friendly data layout is a core HPC skill.

বাংলায়: Memory hierarchy-র নিয়ম: যত ছোট, তত কাছে, তত দ্রুত। Register-এ ~১ cycle, L1-এ ~৪, L2-তে ~১২, L3-তে ~৪০, আর RAM-এ ~২০০ cycle লাগে — মানে RAM থেকে data আনা register-এর চেয়ে প্রায় ২০০ গুণ ধীর। সংখ্যাগুলো order-of-magnitude হিসেবে মুখস্থ রাখুন; পরীক্ষায় ক্রমটা ঠিক রাখাটাই আসল।

4.11 What LINPACK / the Top500 actually measure

  • HPL (High-Performance LINPACK) solves a dense linear system \(Ax = b\) by LU factorisation; the operation count is \(\approx \frac{2}{3}N^3\) FLOP for an \(N \times N\) matrix.
  • The reported number is \(R_{max}\) — the sustained FLOP rate during that solve — always less than \(R_{peak}\).
  • HPL is compute-bound (it does \(O(N^3)\) FLOP on \(O(N^2)\) data), so it flatters machines: 60–80% of peak is normal.
  • Real CFD is memory- and communication-bound; the complementary HPCG benchmark (sparse conjugate gradient) reflects that and typically reaches only 1–3% of peak.
  • Exam phrasing: "The Top500 ranks machines by \(R_{max}\) from the HPL benchmark, published twice a year (June/November)."

5. Command / Syntax / Code Breakdown

There are no shell commands in Lecture 1 itself, but the course already hints at the workflow. Memorise the first-contact commands you will meet on day 2:

Command — login to a cluster (Chapter 4)

ssh <username>@cluster.luis.uni-hannover.de
Field Meaning
ssh Secure shell client
username Your cluster account
cluster... Hostname of the gateway node

Common mistake: typing your password where a public key was expected → repeated Permission denied (publickey). Written-exam tip: state that SSH uses asymmetric encryption.

Command — copy input file to the cluster

scp input.cfg user@cluster:/home/user/sim01/

Command — submit a job (awareness)

sbatch run_simulation.sh

Concept — Linux command structure

command [options/flags] [arguments]

Example: ls -lh /home/user. ls = command, -lh = options (long, human-readable), /home/user = argument.

6. Mandatory Practical Example

Purpose. Demonstrate "Why HPC" with a tiny computational experiment on your own laptop.

Code — save as cpu_test.sh:

#!/bin/bash
# Compare serial vs parallel CPU work
echo "Serial:"
time seq 1 100000000 | awk '{s+=$1} END{print s}'

echo "Parallel (4 chunks):"
time (
  seq 1 25000000        | awk '{s+=$1} END{print s}' &
  seq 25000001 50000000 | awk '{s+=$1} END{print s}' &
  seq 50000001 75000000 | awk '{s+=$1} END{print s}' &
  seq 75000001 100000000| awk '{s+=$1} END{print s}' &
  wait
) | awk '{s+=$1} END{print s}'

Run: bash cpu_test.sh

Expected output

Serial:
5000000050000000
real    0m12.4s

Parallel (4 chunks):
5000000050000000
real    0m3.6s

Step-by-step

  1. seq 1 N generates the integers 1..N.
  2. awk '{s+=$1} END{print s}' sums them.
  3. time reports wall-clock time.
  4. The parallel block launches four background subshells (&) and wait blocks until all finish — process-level parallelism in its simplest form.

Check the math: \(\sum_{k=1}^{10^8} k = \frac{10^8(10^8+1)}{2} = 5000000050000000\) — both versions agree; speed-up \(\approx 12.4/3.6 = 3.4\) on 4 processes (\(E \approx 86\%\)).

বাংলায়: এই ১২-লাইনের script-টাই পুরো কোর্সের দর্শন: কাজ ভাগ করো, একসাথে চালাও, ফল জোড়া দাও। ৪ ভাগে ৩.৪ গুণ গতি (১০০% নয়!) — বাকিটা গেল overhead-এ; এটাই Amdahl-এর প্রথম স্বাদ। পরীক্ষায় "parallelism কীভাবে সময় বাঁচায়" প্রশ্নে এই উদাহরণ + Amdahl উল্লেখ করলেই শক্ত উত্তর।

Real-Life HPC/CFD Meaning. Replace seq with "simulate one cell" and you have a baby version of an OpenMP-parallel CFD loop. On a real cluster the same pattern is mpirun -np 4096 ./solver.

Written Exam Relevance. "Explain how a parallel program saves wall-clock time" — answer with this 4-process model and mention Amdahl's law.

7. Real HPC/CFD Workflow

A textbook day in the life of a CFD researcher:

  1. Edit the input file on your laptop with Vim.
  2. Commit the change with Git.
  3. Push to a GitLab remote.
  4. SSH into the cluster.
  5. git pull the new version.
  6. Compile with make or cmake --build.
  7. Submit the simulation: sbatch run.sh.
  8. Monitor with squeue -u $USER.
  9. Copy results back: rsync -avz cluster:results/ ./local/.
  10. Plot with gnuplot.
  11. Debug with gdb / valgrind if it crashed.
  12. Document outcomes in a Markdown report.

Every step is a chapter of this course. Lecture 1 is the map.

8. Exercises and Solutions

There is no formal exercise sheet for Chapter 1. The companion practical for L01 is E01_WSL — installing the Windows Subsystem for Linux.

E01 (rewritten in simple English). Install Ubuntu inside Windows 10/11 using WSL2, open a Linux terminal, verify with uname -a.

বাংলায়: Windows-এ WSL2 চালু করে উবুন্টু ইনস্টল করো; টার্মিনাল খুলে uname -a দিয়ে যাচাই করো।

Solution (step-by-step):

# In an elevated PowerShell:
wsl --install -d Ubuntu
# After reboot, open "Ubuntu" from the start menu, set username/password, then:
uname -a

Expected line:

Linux DESKTOP-XXXXX 5.15.X-microsoft-standard-WSL2 ... GNU/Linux

Common mistake. Virtualisation disabled in BIOS → WSL refuses to start.

Harder version. Configure WSL to use 4 CPUs and 8 GB RAM via %UserProfile%\.wslconfig:

[wsl2]
processors = 4
memory     = 8GB

Restart with wsl --shutdown, then check inside WSL: nproc → 4; free -h → ≈ 8G.

Marking-scheme answer (5 marks):

  • 1: name WSL2 = a Linux kernel inside Windows.
  • 1: wsl --install -d Ubuntu.
  • 1: virtualisation in BIOS.
  • 1: verify with uname -a.
  • 1: why a Linux env is needed for HPC/CFD.

9. Written Exam Focus

9.1 Short Answers

Q. Define HPC. A. HPC aggregates many CPUs/GPUs across networked nodes so one calculation finishes in hours instead of years — essential for problems with billions of unknowns, e.g. CFD of an aero-engine combustor.

Q. What is the Top500? A. A semi-annual list ranking the 500 fastest publicly-known supercomputers by the LINPACK benchmark in FLOPS.

Q. Why is Linux preferred on HPC clusters? A. Open-source, modular, stable, scriptable through shells, POSIX-compliant — vendors adapt the kernel, users automate every step.

Q. Why are virtual experiments needed alongside real experiments in turbomachinery? A. Real engines are mostly inaccessible to optical diagnostics; CFD gives full 3-D fields of every variable but must be validated against experiments.

9.2 Medium Answers

Q. (8 marks) Compare experiments and CFD simulations as research tools.

A. Experiments give ground truth and are required for validation; but they are expensive, measure only 2–3 quantities simultaneously, and many regions are optically opaque. CFD is cheap to repeat, yields the full field of all variables, and allows design iteration before hardware exists — but needs validation and becomes trustworthy only at high resolution, which forces HPC. The methods are complementary: experiments validate, simulations interpolate and extrapolate.

9.3 Long Answer (12–15 marks)

Q. Discuss the role of HPC in modern CFD-based research on sustainable aviation propulsion.

A. Introduction. HPC is the enabling technology for CFD-driven research on low-emission aero engines.

Main concept. CFD discretises the Navier–Stokes equations on a 3-D mesh; a typical LES of one combustor sector has \(10^8\)–\(10^9\) cells × \(10^5\) time-steps × tens of species equations → \(10^{15}\)–\(10^{18}\) FLOP. Far beyond a workstation; exascale machines (e.g. Frontier at ~1.6 EFLOP/s) do it in days.

Step-by-step example. (i) geometry+mesh, (ii) compile a reactive-flow solver, (iii) parallel job on hundreds of nodes via SLURM, (iv) post-process with gnuplot/ParaView, (v) compare against optical test-rig data. Each step uses Linux, SSH, scripting, version control, parallel libraries — the topics of this course.

Conclusion. HPC turns CFD from a qualitative aid into a predictive design tool, making low-CO₂ engines a realistic target.

9.4 Output Prediction

Q. Order of magnitude of wall-clock time of a DNS with \(10^9\) cells, \(10^5\) steps, 100 ops/cell/step, on a 1 PFLOPS machine?

A. Total ops \(= 10^9 \times 10^5 \times 10^2 = 10^{16}\). Time \(= 10^{16}/10^{15} = 10\) s of pure compute ⇒ in practice 1–10 hours after parallel inefficiencies (memory bandwidth, communication, I/O). Conclusion: HPC required.

9.5 Comparison

Item Workstation HPC cluster
Cores 4–32 \(10^5\)–\(10^7\)
RAM 16–256 GB TB scale
Job control manual SLURM/PBS
Storage local SSD parallel FS (Lustre)
OS Win/macOS/Linux Linux only
User access direct SSH only
Software GUI installs Modules / Spack

9.6 Templates

Definition template: "[Term] is …; it is used to … In HPC/CFD this matters because … Example: …"

Argument template: "The two methods differ in [criteria]. Method A …; Method B …; therefore A is preferable when …"

9.7 Marking Scheme — "Why HPC for CFD?" (5 marks)

  • 1: CFD is computationally expensive.
  • 1: estimate operation count (\(10^{15}\)+).
  • 1: define HPC.
  • 1: name a real cluster (Frontier / El Capitan / JUWELS).
  • 1: link to an engineering goal (aero-engine, climate, drug design).

10. Very Hard Questions

Beginner

  1. What does HPC stand for? → High-Performance Computing.
  2. Name two supercomputers. → El Capitan, Frontier, JUWELS, Aurora.
  3. Which OS runs on HPC clusters? → Linux.
  4. What is a "core"? → One processing unit of a CPU.
  5. What does the Top500 measure? → LINPACK / HPL benchmark performance.

Intermediate

  1. Convert 1 PFLOPS to FLOPS. → \(10^{15}\).
  2. Node vs core. → Node = whole computer; core = execution unit inside a CPU.
  3. Why Linux? → open-source + scriptable + POSIX (any 3).
  4. Simulation vs experiment? → cost vs accessibility vs validation.
  5. Role of the interconnect? → fast inter-node communication.

Hard

  1. FLOPs for an LES of \(5\times10^8\) cells, \(5\times10^4\) steps, 200 ops/cell/step. → \(5\times10^{15}\).
  2. Why does adding cores eventually stop helping? → Amdahl's law.
  3. Why are GPUs on the rise? → higher FLOPS/W, high arithmetic density.
  4. Why is communication latency critical? → strong scaling becomes communication-bound.
  5. Why is power a key Top500 figure? → cooling cost ∝ power; targets push GFLOPS/W.

Very Hard

  1. Show algebraically that speed-up plateaus for fixed serial fraction f. → \(S(N)=1/(f+(1-f)/N) \to 1/f\).
  2. 5% serial code: max speed-up? → \(1/0.05 = 20\times\).
  3. Communication ∝ √N, computation ∝ N per node — when does communication dominate? → when problem size per node shrinks enough that the √N message term exceeds compute (strong-scaling limit).

Deep Integration

  1. Trace a CFD job from Vim edit to final plot, naming each chapter. → Vim (5) → Git (10) → SSH/scp (4) → bash/SLURM (8/15) → C++ build (11/12) → debugging (13) → parallelisation (14) → cluster (15) → gnuplot (9).
  2. Why is Linux "scriptability" more important than "free of charge"? → reproducibility & automation.

Coding/Command

  1. Print the core count. → nproc.
  2. Time a program. → time ./a.out.

Debugging

  1. Why might a parallel program give different results each run? → races / non-deterministic FP reduction order.
  2. Why may speed-up degrade beyond N = 16 on a laptop? → hyperthreads share ALUs / cache contention.

Long Written

  1. (250 words) How does institute research tie Linux, HPC, and sustainable aviation? (Use 9.3 as the model.)

11. Debugging and Mistake Analysis

Mistake Why wrong Correct version Explanation
"Supercomputer = one giant CPU" modern HPC is parallel "many nodes connected by a network" clock frequency stalled ~2005; only parallelism scales
Mixing core and thread threads can share a core core = one instruction stream SMT/HT shares ALUs
"GPUs replace CPUs" they complement heterogeneous nodes have both CPU control + GPU stencil
"HPC = fast desktop" misses parallelism "aggregate parallel performance" definition matters

12. Mini Project for Mastery

Goal: Estimate your laptop's FLOPS and compare to a Top500 entry.

sudo apt install sysbench -y
sysbench cpu --cpu-max-prime=20000 --threads=$(nproc) run

Convert events/s to a rough FLOPS estimate; compare to El Capitan (~1.7 EFLOPS). Expected outcome: your laptop ~\(10^9\), ratio ~\(10^{-9}\) — a billion times slower. A powerful exam-ready statement.

13. Final Chapter Cheat Sheet

Item Memorise
HPC High-Performance Computing — many cores together
CFD Computational Fluid Dynamics
FLOPS Floating-point Ops per Second
\(R_{peak}\) nodes × sockets × cores × clock × FLOP/cycle
Prefixes M(\(10^{6}\)) G(\(10^{9}\)) T(\(10^{12}\)) P(\(10^{15}\)) E(\(10^{18}\))
Top500 top500.org — LINPACK/HPL ranking, June+November
El Capitan #1 (LLNL) ~1.7 EFLOPS
Frontier Oak Ridge exascale
Linux share in Top500 100%
Memory wall latency: reg ~1, L1 ~4, L2 ~12, L3 ~40, RAM ~200 cycles
Common trap adding cores doesn't always help (Amdahl)
Written-exam line "HPC is essential because CFD problems require \(10^{15}\)+ ops which exceed any single workstation."

14. Mock Exam — Four Levels

Level 1 — Basic (definitions & syntax)

Q1. Expand and define FLOPS.

Solution: Floating-point Operations Per Second — the rate of floating-point arithmetic a machine sustains.

Q2. Order these by size: TFLOPS, GFLOPS, EFLOPS, PFLOPS.

Solution: GFLOPS (\(10^9\)) < TFLOPS (\(10^{12}\)) < PFLOPS (\(10^{15}\)) < EFLOPS (\(10^{18}\)).

Q3. What benchmark ranks the Top500?

Solution: LINPACK (HPL) — dense linear-system solve, reporting \(R_{max}\).

Q4. Name the four hardware levels from core to cluster.

Solution: core → socket/CPU → node → (rack →) cluster.

Q5. What is a node?

Solution: One complete computer in the cluster (CPUs, RAM, NIC), connected to others via the interconnect.

Level 2 — Intuitive (predict / explain why)

Q1. Why did CPU clock speeds stop rising around 2005, and what replaced frequency scaling?

Solution: Power density (heat) grows superlinearly with clock; cooling became impossible → parallelism (more cores) replaced frequency scaling.

Q2. A "1 TB" disk shows 931 GiB in Linux. Where did the space go?

Solution: Nowhere — units differ: \(10^{12}\) B ÷ \(2^{30}\) B/GiB \(= 931\) GiB. Marketing uses powers of 10, OS uses powers of 2.

Q3. Two machines: A = few cores at 5 GHz, B = many cores at 2 GHz with equal total FLOPS. Which is better for CFD, and why?

Solution: B in general — CFD is mesh-parallel and memory-bandwidth-hungry; many cores with wide memory beat few fast cores. (A wins only for stubbornly serial workloads — Amdahl.)

Q4. Why does doubling cores rarely halve runtime?

Solution: Serial fraction (Amdahl), communication/synchronisation overhead, and memory-bandwidth saturation.

Q5. Why does the Top500 also publish power (MW) per machine?

Solution: Energy is the real operating constraint — GFLOPS/W (Green500) decides feasibility; an exascale machine draws ~20–30 MW.

Level 3 — Hard (exam level)

Q1. (8 marks) Compute \(R_{peak}\) for one node: 2 sockets × 24 cores × 2.4 GHz, AVX-512 with FMA (32 DP FLOP/cycle/core). Then for 100 nodes, in PFLOPS.

Solution: Per node: \(2 \times 24 \times 2.4\times10^9 \times 32 = 3.686\times10^{12}\) → 3.69 TFLOPS. 100 nodes: \(0.369\) PFLOPS. বাংলা ইঙ্গিত: সূত্রটা গুণফল মাত্র — সব ইউনিট ঠিক রেখে (GHz → \(10^{9}\)) পরপর গুণ করো; FLOP/cycle = vector width × 2 (FMA)।

Q2. (8 marks) A solver does \(10^{14}\) FLOP per timestep and sustains 5% of the 0.369 PFLOPS machine above. Time per step? For \(10^4\) steps?

Solution: Sustained \(= 0.05 \times 3.69\times10^{14} = 1.84\times10^{13}\) FLOP/s. Per step: \(10^{14}/1.84\times10^{13} \approx 5.4\) s. Total: \(5.4\times10^4\) s ≈ 15 h. বাংলা ইঙ্গিত: "sustained %" না দিলে peak ধরে নিও না — বাস্তবে 1–10%; পরীক্ষাও ঠিক এই ফাঁদটাই পাতে।

Q3. (8 marks) Machine balance: a node has 3.69 TFLOPS peak and 400 GB/s memory bandwidth. Compute bytes/FLOP, and decide whether a stencil code needing 1 byte/FLOP is compute- or memory-bound there.

Solution: Balance \(= 400\times10^9 / 3.69\times10^{12} \approx 0.11\) bytes/FLOP. The code needs 1 byte/FLOP \(\gg\) 0.11 → memory-bound: the CPU starves; effective performance ≈ bandwidth-limited \(400\times10^9 \times 1\) FLOP/byte... i.e. ~0.4 TFLOPS (11% of peak). বাংলা ইঙ্গিত: machine balance < code demand ⇒ memory-bound — CFD-র প্রায় সব stencil-ই এই দলে; "১০% peak-ই স্বাভাবিক" — এই বাক্যটা লিখো।

Q4. (10 marks) Using the memory-hierarchy latencies (L1 ~4, RAM ~200 cycles), explain quantitatively why blocking/tiling a matrix kernel can give ~10× speed-up.

Solution: Unblocked: most accesses miss cache → ~200-cycle stalls dominate; effective ~1 op per ~200 cycles of waiting in the worst case. Blocked: working set fits in L1/L2 → ~4–12 cycle access; ratio 200/12 ≈ 16× fewer stall cycles, realised as ~an order of magnitude — locality, not arithmetic, sets the speed. বাংলা ইঙ্গিত: সংখ্যা দুটো (4 বনাম 200) উত্তরে রাখলেই "quantitatively" শর্ত পূরণ — অনুপাতই গল্পের নায়ক।

Q5. (10 marks) Estimate (order of magnitude) how long the serial cpu_test.sh sum of \(10^8\) numbers SHOULD take if the machine ran at 1 GFLOPS effective, and explain why the measured 12 s is far larger.

Solution: \(10^8\) additions / \(10^9\) FLOP/s \(= 0.1\) s ideal. Measured 12 s ⇒ ~120× overhead: the pipeline spawns processes, formats/parses ASCII decimal text through awk (dozens of cycles per char), and crosses a pipe — I/O and parsing dominate, not arithmetic. Lesson: real performance is set by data movement, the recurring HPC theme. বাংলা ইঙ্গিত: "কেন এত ধীর" প্রশ্নের উত্তর কখনোই "কম্পিউটার ধীর" নয় — text parsing/process overhead-এর মতো নির্দিষ্ট খরচ ধরিয়ে দাও।

Level 4 — Beyond the lecture (transfer + coding)

Q1. Write a portable bash snippet printing a one-line summary of this machine: hostname, #cores, RAM (GiB), kernel version.

Solution: 

printf "%s: %s cores, %s GiB RAM, kernel %s\n" \
  "$(hostname)" "$(nproc)" \
  "$(awk '/MemTotal/ {printf "%.1f", $2/1048576}' /proc/meminfo)" \
  "$(uname -r)"
/proc/meminfo reports KiB → divide by \(1024^2\) for GiB; everything is standard on any Linux node. বাংলা ইঙ্গিত: /proc-ফাইলগুলোই Linux-এর "নিজের সম্পর্কে সত্য" — awk দিয়ে পড়াটা দুই chapter-এর মেলবন্ধন।

Q2. You must choose between 1 node with 128 cores or 8 nodes with 16 cores each (same total). For (a) an embarrassingly parallel parameter sweep and (b) a tightly-coupled CFD halo-exchange solver, which do you pick and why?

Solution: (a) Either works — no communication; pick by availability/queue time (8 small nodes often schedule faster). (b) The single 128-core node: halo exchange stays in shared memory/NUMA instead of crossing the network — lower latency, no interconnect bottleneck (Ch 14 surface-to-volume logic). বাংলা ইঙ্গিত: প্রশ্নের ধরন চেনো: communication-হীন কাজ topology-উদাসীন; coupled কাজ network এড়াতে চায়।

Q3. A vendor claims "our 2 EFLOPS machine will run your code 2000× faster than your 1 TFLOPS workstation." Give two quantitative reasons this is misleading.

Solution: (1) Amdahl: with even 1% serial fraction, \(S_{max} = 100\) — never 2000×, however many cores. (2) The claim compares peak values: your code's sustained fraction (often 1–10% of peak, memory-bound) differs per machine; LINPACK ratios don't transfer to stencil codes. (Bonus: strong-scaling communication overheads grow with node count.) বাংলা ইঙ্গিত: মার্কেটিং-দাবি ভাঙার দুই হাতুড়ি সবসময়: Amdahl আর peak-বনাম-sustained।

Q4. Estimate the energy (kWh) of a 24 h job on 50 nodes at 500 W/node, and the CO₂ (at 400 g/kWh). Should the course's bash/awk skills factor into "green computing"? One concrete example.

Solution: \(50 \times 0.5\,\text{kW} \times 24\,\text{h} = 600\) kWh → \(600 \times 0.4 = 240\) kg CO₂. Yes: profiling and scripting prevent waste — e.g. an awk check that stops a diverged run after 10 min (residual = nan in the log) saves the remaining 23.8 h × 50 nodes ≈ 595 kWh of garbage compute. বাংলা ইঙ্গিত: গণনাটা সহজ (kW × h), কিন্তু "স্ক্রিপ্টিং = সবুজ HPC" সংযোগটা দেখাতে পারাই out-of-topic প্রশ্নের আসল উত্তর।

End of Chapter 1.