Chapter 13: Debugging — gdb, valgrind, sanitizers¶

Source slides: V11_debugging.pdf. Exercise: E11_debugging.pdf (Mapper2D task01.cpp). Lecture examples: lecture_examples/debug01.cpp..debug06.cpp.

1. Chapter Overview¶

In real CFD code, bugs appear as: NaN residuals after 5 days of compute, "segmentation fault" on rank 12 only, memory leaks that grow with iteration count, off-by-one indices that produce silently wrong physics. The two heavy-weight tools are gdb (the GNU Debugger — interactive line-by-line execution, breakpoints, watchpoints, backtraces) and valgrind (memory-error detector). Modern alternatives are AddressSanitizer / UBSan / ThreadSanitizer (-fsanitize=…).

Why it matters in HPC/CFD: a CFD job uses thousands of CPU-hours. A wrong-index bug found by a sanitizer in 30 seconds saves tens of thousands of euros.

What the examiner asks (very common):

"What does gdb do?"
"Compile with debug info — what flag?"
"Find the bug in this code." (off-by-one, NULL deref, infinite loop, leak)
"Explain break, run, next, step, print, continue, watch, backtrace."
"Difference between next and step."
"What does valgrind detect?"
"Predict the output of debug05.cpp."

What you must master for top grade:

Compile with -g (and ideally -O0 -g) for debug.
The 12 most-used gdb commands.
The 6 lecture bugs and how to find each.
valgrind's --leak-check=full --track-origins=yes.
Sanitizers as the modern fast alternative.

2. Basics from Zero¶

A debugger lets you stop a running program, look at variables, step one line at a time, and continue. To use it, compile with -g:

g++ -O0 -g debug01.cpp -o debug01
gdb ./debug01

Inside gdb:

(gdb) break 8       # set breakpoint at line 8
(gdb) run           # start the program
(gdb) print sum     # show variable
(gdb) next          # next line (stay in same function)
(gdb) step          # step into function call
(gdb) continue      # run until next breakpoint
(gdb) backtrace     # call stack
(gdb) quit

বাংলায়: Debugger মানে এমন একটা টুল যেটা দিয়ে চলন্ত প্রোগ্রামকে মাঝপথে থামিয়ে ভেতরের variable-গুলোর মান দেখা যায়। gdb ব্যবহার করতে হলে আগে -g flag দিয়ে compile করতে হবে — না হলে gdb source code-এর লাইন নম্বর চিনবে না। পরীক্ষায় প্রায়ই জিজ্ঞেস করে কোন flag লাগে আর breakpoint কী করে, তাই এই দুটো জিনিস মুখস্থ রাখো।

Memory bugs (use of freed memory, leaks, uninitialised reads, double-free) are best caught by valgrind ./prog — it runs the program through a virtual machine and reports every problem.

বাংলায়: valgrind প্রোগ্রামটাকে একটা virtual machine-এর ভেতরে চালায় এবং প্রতিটা memory operation-এর হিসাব রাখে — কোথায় malloc হলো, কোথায় free হলো না, কোথায় array-র বাইরে লেখা হলো। gdb দিয়ে leak ধরা যায় না, কারণ leak কোনো crash ঘটায় না; এজন্যই valgrind আলাদা টুল হিসেবে দরকার। পরীক্ষায় "কোন টুল কোন bug ধরে" এই প্রশ্নটা খুবই common।

Real-life analogy. A debugger is a video player for code: pause, step frame-by-frame, rewind variables. valgrind is an x-ray showing memory traffic.

Real-life HPC use. A CFD solver crashes after 200 iterations. Compile with -g, run under gdb, set a breakpoint at iteration 200's residual computation, inspect the field — find an out-of-bound index. Or compile with -fsanitize=address -g and let the sanitizer print the offending line.

What if you misunderstand? You compile with -O3 and no -g → gdb shows <optimized out> for variables. Or you forget valgrind and ship a leak that grows by 1 MB/iteration → 24 GB after 24 000 iterations.

বাংলায়: সবচেয়ে বড় ভুল হলো optimisation চালু রেখে debug করা। -O3 দিলে compiler অনেক variable মুছে ফেলে বা loop ভেঙে নতুন করে সাজায়, তখন gdb-তে "optimized out" দেখায় আর source-এর লাইনের সাথে execution মেলে না। তাই নিয়ম একটাই: debug build মানে -O0 -g, release build মানে -O3। এটা পরীক্ষার একটা প্রিয় trap প্রশ্ন।

3. Hard English Made Easy¶

Hard Term	Simple English	বাংলা	Example
Debugger	Tool to inspect running program	চলমান প্রোগ্রাম পরীক্ষা যন্ত্র	gdb
Breakpoint	Pause point	বিরতি বিন্দু	`break 8`
Watchpoint	Pause when variable changes	পরিবর্তন পর্যবেক্ষণ	`watch i`
Backtrace / call stack	Chain of calls	ফাংশন কল চেইন	`bt`
Frame	One level of stack	এক স্তরের স্ট্যাক	`frame 2`
Step	Step into call	ফাংশনে ঢোকা	`step`
Next	Step over call	কল সম্পূর্ণ চালানো	`next`
Continue	Run until next stop	পরবর্তী বিরতি পর্যন্ত	`cont`
Memory leak	Allocated, never freed	মেমরি লিক	`malloc` w/o `free`
Use after free	Access already freed memory	ফ্রি করার পর ব্যবহার	UAF
Double free	Free twice	দ্বিগুণ ফ্রি	crash
Uninitialised read	Use before set	অপ্রাথমিক পঠন	UB
Buffer overflow	Write past array end	অ্যারের সীমা অতিক্রম	classic CVE
Segmentation fault	Bad memory access	মেমরি অ্যাক্সেস ত্রুটি	`*nullptr`
Sanitizer	Runtime checker	রানটাইম পরীক্ষক	`-fsanitize=address`
Valgrind	Memory checker (slow)	মেমরি পরীক্ষক	`valgrind ./a`
Optimisation level	`-O0..-O3`	অপটিমাইজেশন	`-O0` for debug
Symbol info	Source/line metadata	সিম্বল মেটাডাটা	`-g`
Core dump	Memory image at crash	ক্র্যাশ মেমরি ছবি	`core` file

4. Deep Theory Explanation¶

4.1 Compile for debugging¶

g++ -O0 -g -Wall -Wextra debug01.cpp -o debug01

-g embeds DWARF debug symbols.
-O0 disables optimisation so variables match the source.
-fno-omit-frame-pointer (sometimes) keeps the stack readable.

বাংলায়: এখানে -g flag binary-র ভেতরে DWARF নামের একটা টেবিল ঢুকিয়ে দেয়, যেটা machine code-এর প্রতিটা address-কে source file-এর লাইন নম্বরের সাথে মিলিয়ে দেয়। -O0 নিশ্চিত করে compiler কোনো variable মুছে ফেলেনি — তাই gdb-তে যা দেখবে, source-এ ঠিক তা-ই আছে। পরীক্ষায় "compile flag for debugging" প্রশ্ন এলে উত্তর সবসময়: g++ -O0 -g।

4.2 gdb command catalogue¶

Cmd	Action
`gdb ./prog`	start gdb
`run [args]`	start program
`break <line>` / `b 8`	breakpoint by line
`break main`	by function
`break file:line`	qualified
`info breakpoints`	list
`delete N`	remove break point
`disable N`, `enable N`	toggle
`next` / `n`	step over
`step` / `s`	step into
`finish`	run to end of function
`continue` / `c`	run to next stop
`until <line>`	run until line
`print x` / `p x`	show value
`print *arr@10`	array slice
`display x`	auto-print on each stop
`info locals`	all locals
`info args`	function args
`watch <expr>`	break when expr changes
`rwatch / awatch`	read / read-or-write
`backtrace` / `bt`	call stack
`frame N`	switch frame
`up`, `down`	move up/down stack
`list` / `l`	show source
`layout src`	TUI source view
`set variable x=5`	modify variable
`quit` / `q`	exit

বাংলায়: এই টেবিল থেকে পরীক্ষার জন্য সবচেয়ে জরুরি জোড়া হলো step বনাম next: step ফাংশনের ভেতরে ঢোকে, next পুরো ফাংশন কলটা এক ধাপে চালিয়ে দেয়। আর watch হলো এমন breakpoint যেটা লাইনে নয়, কোনো variable-এর মান বদলালে থামে — infinite loop বা রহস্যময় value-পরিবর্তন ধরার জন্য দারুণ অস্ত্র। কমপক্ষে break, run, print, bt, watch — এই পাঁচটা কমান্ড syntax-সহ মুখস্থ রাখো।

4.3 valgrind essentials¶

valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes ./debug06

Reports:

Definitely lost = leak (debug06.cpp lacks free).
Indirectly lost = leak via leaked pointer.
Possibly lost = pointer from middle of block.
Still reachable = OK at exit, but tidy up.

valgrind's tools: memcheck (default), helgrind (race conditions), cachegrind (cache profile), callgrind (call profile), massif (heap profile).

বাংলায়: valgrind-এর রিপোর্টে চার রকম leak আছে, কিন্তু আসল অপরাধী হলো "definitely lost" — মানে ওই memory block-এর কোনো pointer আর কোথাও নেই, ফেরত পাওয়ার কোনো উপায় নেই। "Still reachable" তুলনায় নিরীহ: exit-এর সময়েও pointer ছিল, শুধু free করা হয়নি। পরীক্ষায় output দেখিয়ে জিজ্ঞেস করতে পারে কোন লাইনটা আসল leak — তখন "definitely lost"-এর নিচের backtrace-এ malloc-এর লাইন নম্বরটাই উত্তর।

Anatomy of a valgrind report (every line decoded — this is what the examiner wants you to read fluently):

==4242== Invalid write of size 4                  ◄─ wrote 4 bytes (one int) where
==4242==    at 0x10916B: main (debug01.cpp:6)        you were not allowed to
==4242==                          └──────┬──────┘
==4242==                  WHERE the bad write happened (needs -g!)
==4242==  Address 0x4a4b054 is 0 bytes after      ◄─ the address sits 0 bytes PAST
==4242==  a block of size 20 alloc'd                 the end of a 20-byte block
==4242==    at 0x4848899: malloc (vg_replace_...)    (20 B = 5 ints → arr[5] access!)
==4242==    by 0x109151: main (debug01.cpp:4)     ◄─ WHERE that block was allocated
==4242==
==4242== HEAP SUMMARY:
==4242==     in use at exit: 20 bytes in 1 blocks ◄─ something was never freed
==4242==   total heap usage: 1 allocs, 0 frees    ◄─ allocs != frees → leak exists
==4242==
==4242== 20 bytes in 1 blocks are definitely lost ◄─ TRUE LEAK: no pointer to the
==4242==    at 0x4848899: malloc (...)               block survives; the backtrace
==4242==    by 0x109151: main (debug06.cpp:7)     ◄─ names the line you must fix

The ==4242== prefix is simply the process ID. "Invalid write of size 4" = a 4-byte (one int) out-of-bounds or use-after-free write; "0 bytes after a block of size 20" decodes to: the program wrote at index 5 of a 5-int array.

4.4 Sanitizers (modern preferred)¶

g++ -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer prog.cpp -o prog
./prog

AddressSanitizer (ASan) catches: heap/stack/global buffer overflow, use-after-free, use-after-return, double-free. UBSan catches: signed overflow, divide-by-zero, NULL deref, misaligned load. TSan catches: data races (-fsanitize=thread).

Cost: ~2× slower at runtime — vs valgrind's ~30×.

বাংলায়: Sanitizer হলো compile-time instrumentation: compiler নিজেই প্রতিটা memory access-এর আগে-পরে চেক বসিয়ে দেয়, তাই মাত্র ~2 গুণ ধীর হয় — valgrind-এর ~30 গুণের তুলনায় অনেক দ্রুত। দাম একটাই: recompile করতে হবে, যেটা valgrind-এ লাগে না। আরেকটা মোক্ষম পার্থক্য মনে রাখো: stack array-র overflow (যেমন debug01-এর numbers[5]) valgrind ধরতে পারে না, কিন্তু ASan ধরে — পরীক্ষায় tool-choice প্রশ্নে এটাই trick।

4.5 The 6 lecture bugs¶

File	Bug	Detection
`debug01.cpp`	Off-by-one (loop `i<=5` for size-5 array)	gdb watch + sanitizer
`debug02.cpp`	None — clean program (used to demo `step` into `add`)	step / next
`debug03.cpp`	NULL pointer dereference	gdb run → segfault, `bt`
`debug04.cpp`	Infinite loop (count never increments)	gdb Ctrl-C, `info locals`
`debug05.cpp`	Logical bug: `is_prime` returns `true` for `num<2` because of double negation	gdb `b 24`, step in
`debug06.cpp`	Memory leak (malloc'd but never freed)	valgrind

বাংলায়: এই ছয়টা lecture-bug আসলে ছয় শ্রেণির bug-এর প্রতিনিধি: সীমা ছাড়ানো index (debug01), NULL pointer (debug03), infinite loop (debug04), logic ভুল (debug05), আর memory leak (debug06)। পরীক্ষক প্রায়ই এর একটা হুবহু বা সামান্য বদলে দিয়ে বলে "find the bug" — তাই প্রতিটার লক্ষণ (segfault? hang? ভুল উত্তর? RAM বাড়ছে?) আর কোন টুল দিয়ে ধরবে, দুটোই একসাথে শিখে রাখো।

4.6 Diagrams from V11¶

gdb commands flow diagram: edit → compile -g → gdb → run → breakpoint → step → print → continue → quit.
valgrind diagram: instrument every memory op → report at exit.

4.7 The mathematics of bug hunting — bisection¶

Debugging has one genuinely beautiful piece of mathematics: binary search over history (or code). Suppose a bug is hidden in one of $N$ candidates (commits, lines, time-steps). Each test ("does the bug appear?") splits the candidate set in half. After $k$ tests, at most $N/2^k$ candidates remain, so we need

\[ \frac{N}{2^k} \le 1 \quad\Longrightarrow\quad k \ge \log_2 N \quad\Longrightarrow\quad k_{\min} = \lceil \log_2 N \rceil . \]

Compare with linear search, which needs on average $N/2$ tests and in the worst case $N$.

Worked example 1 — git bisect over 1024 commits. A regression appeared somewhere between tag v1.0 (good) and HEAD (bad), 1024 commits apart.

\[ k = \lceil \log_2 1024 \rceil = \lceil 10 \rceil = 10 \text{ tests.} \]

Step-by-step shrinkage: $1024 \to 512 \to 256 \to 128 \to 64 \to 32 \to 16 \to 8 \to 4 \to 2 \to 1$. That is exactly 10 halvings. If each compile-and-test cycle takes 6 minutes, bisection costs $10 \times 6 = 60$ min; linear search would average $512 \times 6 = 3072$ min ≈ 51 hours.

git bisect start
git bisect bad HEAD          # current version is broken
git bisect good v1.0         # this old tag was fine
# git checks out the middle commit; you compile + test, then:
git bisect good              # or: git bisect bad
# ... repeat 10 times; git prints the first bad commit ...
git bisect reset

Worked example 2 — bisecting source lines. A mysterious crash is caused by one of $N = 50\,000$ lines in a translation unit. Commenting out half the suspect region per test: $\lceil \log_2 50\,000 \rceil = \lceil 15.61 \rceil = 16$ tests suffice (since $2^{15} = 32\,768 < 50\,000 \le 65\,536 = 2^{16}$).

বাংলায়: Bisection-এর মূল কথা: প্রতিটা টেস্টে সন্দেহভাজনের সংখ্যা অর্ধেক হয়ে যায়, তাই N-টা commit-এর মধ্যে bug খুঁজতে লাগে মাত্র log2(N)-এর ceiling-সংখ্যক টেস্ট — 1024 commit-এ মাত্র 10টা। পরীক্ষায় সংখ্যা দিয়ে জিজ্ঞেস করলে আগে দেখো N কি 2-এর power; না হলে পরের 2-এর power পর্যন্ত উঠে ceiling নাও। git bisect ঠিক এই গণিতটাই automate করে।

4.8 Memory-error taxonomy — which tool catches what¶

                            MEMORY ERRORS
                                  │
     ┌──────────────┬─────────────┼──────────────┬──────────────────┐
     ▼              ▼             ▼              ▼                  ▼
 ┌────────┐  ┌─────────────┐ ┌──────────┐ ┌─────────────┐  ┌───────────────┐
 │  LEAK  │  │ USE-AFTER-  │ │ DOUBLE-  │ │ OUT-OF-     │  │ UNINITIALIZED │
 │        │  │    FREE     │ │  FREE    │ │ BOUNDS      │  │     READ      │
 │ malloc │  │ read/write  │ │ free()   │ │ read/write  │  │ use a value   │
 │ without│  │ a block     │ │ called   │ │ past array  │  │ before it is  │
 │ free   │  │ already     │ │ twice on │ │ end (heap   │  │ ever set      │
 │        │  │ freed       │ │ same ptr │ │ or stack)   │  │               │
 └───┬────┘  └──────┬──────┘ └────┬─────┘ └──────┬──────┘  └───────┬───────┘
     ▼              ▼             ▼              ▼                 ▼
 valgrind       ASan:         ASan:        heap: valgrind     valgrind
 --leak-check   "heap-use-    "double-     "Invalid write     "Conditional
 =full →        after-free"   free" /      of size 4" AND     jump depends on
 "definitely    valgrind:     valgrind:    ASan; stack:       uninitialised
 lost";  also   "Invalid      "Invalid     ONLY ASan          value(s)" with
 LeakSanitizer  read ...      free()"      "stack-buffer-     --track-origins
 (in ASan)      free'd"                    overflow"          =yes;  NOT ASan

Decision table (memorise the two asymmetries marked in bold):

Error class	Typical symptom	gdb	valgrind memcheck	`-fsanitize=address`
Memory leak	RAM grows, OOM after hours	no	yes — "definitely lost"	yes (LeakSanitizer at exit)
Use-after-free	garbage values, late crash	only after crash (`bt`)	yes — invalid read/write in free'd block	yes — "heap-use-after-free"
Double free	`glibc` abort	backtrace at abort	yes — "Invalid free()"	yes — "double-free"
Heap out-of-bounds	silent corruption, crash far away	hard	yes — "Invalid write of size 4"	yes — "heap-buffer-overflow"
Stack out-of-bounds	silent corruption	hard	no — valgrind misses stack arrays	yes — "stack-buffer-overflow"
Uninitialised read	nondeterministic results	no	yes — "Conditional jump … uninitialised"	no — needs valgrind (or MSan)
NULL dereference	immediate SIGSEGV	yes — `run` then `bt`	yes	yes

Rule of thumb: crash location → gdb; leak or uninitialised read → valgrind; out-of-bounds (especially stack) and use-after-free during development → ASan, because it is only ~2× slower.

বাংলায়: এই টেবিলের দুটো অসামঞ্জস্যই পরীক্ষার আসল প্রশ্ন: valgrind stack array-র out-of-bounds ধরতে পারে না (শুধু heap পারে), আর ASan uninitialised read ধরতে পারে না (ওটার জন্য valgrind লাগে)। বাকি সব error দুটো টুলই ধরে, তখন বেছে নাও গতি (ASan ~2x) বনাম recompile-না-করার সুবিধা (valgrind) অনুযায়ী। উত্তরে সবসময় টুলের সাথে তার output-এর exact বাক্যাংশ ("definitely lost", "stack-buffer-overflow") লিখলে পুরো নম্বর পাবে।

4.9 Annotated gdb session — catching an off-by-one live¶

The buggy program (classic debug01-style off-by-one):

// buggy.cpp
#include <iostream>
int main() {
    int numbers[5] = {1, 2, 3, 4, 5};   // valid indices: 0..4
    int sum = 0;
    for (int i = 0; i <= 5; ++i)        // BUG: i<=5 also reads numbers[5]
        sum += numbers[i];
    std::cout << "sum = " << sum << "\n";
    return 0;
}

Full transcript, every step annotated:

$ g++ -O0 -g buggy.cpp -o buggy   ◄── -g embeds line info, -O0 keeps
                                      every variable inspectable
$ gdb ./buggy
(gdb) break 6                     ◄── breakpoint on the loop BODY line
Breakpoint 1 at 0x1175: file buggy.cpp, line 6.
(gdb) run                         ◄── start; gdb stops at first hit
Breakpoint 1, main () at buggy.cpp:6
6               sum += numbers[i];
(gdb) display i                   ◄── auto-print i at EVERY stop
1: i = 0
(gdb) display sum
2: sum = 0
(gdb) continue                    ◄── resume; stops again next iteration
Breakpoint 1, main () at buggy.cpp:6
1: i = 1                2: sum = 1
(gdb) continue                    ◄── keep going: i=2,3,4 look fine...
Breakpoint 1, main () at buggy.cpp:6
1: i = 5                          ◄── !! loop ran with i=5 (i<=5)
2: sum = 15
(gdb) print numbers[i]            ◄── read past the end:
$1 = 32767                        ◄── garbage — OUT OF BOUNDS
(gdb) quit

The transcript shows the exam-ready story: display auto-prints loop state at every stop; the final iteration exposes i==5 reading numbers[5] — one past the end of a 5-element array. Fix: i < 5.

বাংলায়: gdb-session প্রশ্নে পুরো নম্বর পাওয়ার রেসিপি: (১) -O0 -g দিয়ে compile, (২) সন্দেহের লাইনে break, (৩) display i আর display sum — প্রতি স্টপে নিজে নিজে প্রিন্ট হবে, (৪) continue চাপতে চাপতে যেই iteration-এ অস্বাভাবিক মান (i=5, garbage value) — সেটাই bug। উত্তরে এই চারটা ধাপ ক্রমে লিখলেই হলো।

5. Command / Syntax / Code Breakdown¶

`g++ -g -O0 file.cpp -o exe`¶

Build with debug info, no optimisation.

`gdb ./exe`¶

Start interactive debugger.

`(gdb) break 8`¶

Pause at line 8 in current file.

`(gdb) break add`¶

Pause at start of function add.

`(gdb) run`¶

Start (or restart) the program.

`(gdb) next` / `step`¶

Step over / step into.

`(gdb) print x` / `(gdb) p *arr@5`¶

Show variable / array slice.

`(gdb) watch i`¶

Pause when i changes.

`(gdb) backtrace`¶

Show stack at current pause.

`valgrind --leak-check=full ./exe`¶

Memory error report.

`-fsanitize=address`¶

Compiler flag enabling ASan.

6. Mandatory Practical Examples¶

Example 6.1 — Off-by-one (debug01.cpp)¶

int numbers[5] = {1,2,3,4,5};
int sum = 0;
for (int i = 0; i <= 5; ++i)          // BUG: i<=5 reads numbers[5]
    sum += numbers[i];

Detection

g++ -O0 -g debug01.cpp -o d1
gdb ./d1
# (gdb) b 8 ; run ; display sum ; display i ; n ... until i=5

Or with sanitizer:

g++ -O1 -g -fsanitize=address debug01.cpp -o d1
./d1
# ==12345==ERROR: AddressSanitizer: stack-buffer-overflow ... at numbers[5]

Fix: for (int i = 0; i < 5; ++i)

Example 6.2 — Trace add function (debug02.cpp)¶

int add(int a, int b) { return a + b; }

int main(){
    int a, b;
    std::cin >> a; std::cin >> b;
    int result = add(a,b);
    std::cout << "Addition = " << result << "\n";
}

gdb ./d2
(gdb) b 19          # the call
(gdb) run
3
4
(gdb) s             # step into add
(gdb) info locals
(gdb) n
(gdb) c

Example 6.3 — NULL deref (debug03.cpp)¶

int *ptr = NULL;
std::cout << "Value at ptr = " << *ptr << std::endl;    // segfault

gdb ./d3
(gdb) run
Program received signal SIGSEGV, Segmentation fault.
0x... in main () at debug03.cpp:6
(gdb) bt
#0 ...

Fix: Initialise the pointer to a valid address (and test for null).

Example 6.4 — Infinite loop (debug04.cpp)¶

int count = 0;
int arr[6] = {1,2,3,4,5,6};
while (count < 5) {
    arr[count] = 23;          // BUG: count never increments → infinite loop
}

gdb ./d4
(gdb) run            # hangs
^C                   # Ctrl-C
(gdb) info locals
count = 0
arr = {23,2,3,4,5,6}
(gdb) watch count    # would never trigger; the write goes to arr only

Fix: ++count; inside the loop.

Example 6.5 — Logical bug (debug05.cpp)¶

bool is_prime(int num){
    bool prime = true;
    if (num < 2) return !prime;     // returns false for num<2
    else {
        for (int i = 2; i < num; i++)
            if (num % i == 0) prime = false;
        return prime;
    }
}

int main(){
    for (int i = 0; i <= 30; i++)
        if (is_prime(i + 1)) std::cout << i+1 << " ";
}

The lecture's point: step through with gdb and verify each return value. Note is_prime(1): 1<2 → returns !true = false, so 1 is correctly excluded. If a variant prints 1, the bug would be return prime; (missing !). Also note the loop tests up to i+1 = 31.

Example 6.6 — Memory leak (debug06.cpp)¶

#include <iostream>
#include <stdlib.h>
int main() {
    int *arr = (int*)malloc(5 * sizeof(int));
    // never free
    return 0;
}

Detection

valgrind --leak-check=full ./d6
==12345== HEAP SUMMARY:
==12345==     in use at exit: 20 bytes in 1 blocks
==12345==   total heap usage: 1 allocs, 0 frees, 20 bytes allocated
==12345==
==12345== 20 bytes in 1 blocks are definitely lost in loss record 1 of 1
==12345==    at 0x...: malloc (vg_replace_malloc.c:...)
==12345==    by 0x...: main (debug06.cpp:7)

Fix: free(arr); (C) or new[]+delete[] (C++) — better, std::vector<int> (RAII).

Example 6.7 — E11 Mapper2D (task01.cpp)¶

int xyToPos(int x, int y, int sx, int sy)  { return x + (sx-1)*y; }   // BUG: should be sx*y
int posToX (int pos, int sx, int sy)       { return pos % (sx-1); }   // BUG
int posToY (int pos, int sx, int sy)       { return pos / sy; }       // BUG: should be /sx

When you run printMap(5,5) you'd expect a 5×5 grid 0…24 but get repeats / wrong indices.

Detection (the lecture's intended workflow)

g++ -O0 -g task01.cpp -o map
gdb ./map
(gdb) b xyToPos
(gdb) run
(gdb) p x
(gdb) p y
(gdb) p sx
(gdb) p y*sx + x       # what the right formula gives
(gdb) p y*(sx-1) + x   # what the buggy formula gives

Fix

int xyToPos(int x, int y, int sx, int sy) { return x + sx*y; }
int posToX (int pos, int sx, int sy)      { return pos % sx; }
int posToY (int pos, int sx, int sy)      { return pos / sx; }

Real-Life HPC/CFD Meaning. Index mappers are at the heart of every structured-grid CFD code; an off-by-one corrupts the entire solution.

Written Exam Relevance. "Find the bug and write the fix" with a line-by-line gdb walkthrough is a top-mark question.

বাংলায়: Mapper2D হলো এই কোর্সের সবচেয়ে সম্ভাব্য debugging প্রশ্ন। মূল সূত্র: 2D→1D index মানে pos = y·sx + x (row-major) — প্রতি সারিতে sx-টা cell। উল্টোপথে x = pos % sx, y = pos / sx। gdb-তে দুটো formula পাশাপাশি print করে দেখানোই lecture-এর শেখানো কৌশল।

7. Real HPC/CFD Workflow¶

# 1. Compile two flavours
g++ -O3 -march=native -DNDEBUG -fopenmp solver.cpp -o solver_release
g++ -O0 -g -fsanitize=address,undefined -fopenmp solver.cpp -o solver_debug

# 2. Reproduce on a small case
./solver_debug small.cfg
# AddressSanitizer prints first error → fix.

# 3. valgrind for leaks
valgrind --leak-check=full --track-origins=yes ./solver_debug small.cfg

# 4. After fix, run release on cluster
sbatch run.sbatch

For MPI:

mpirun -np 4 xterm -e gdb --args ./solver_debug small.cfg
# attaches gdb to each rank

Tip: enable core dumps with ulimit -c unlimited; after a crash, gdb ./prog core shows the spot.

8. Exercises and Solutions¶

E11 — Mapper2D — see 6.7.

Marking scheme (10 marks)

1 compile with -g.
1 launch gdb.
1 set breakpoint inside xyToPos.
1 step / print arguments.
2 spot the wrong formula(s).
2 propose corrected formula(s).
1 verify with printMap(5,5).
1 final clean output.

Common mistakes

Compiling with -O2 (variables optimised out, gdb shows <optimized out>).
Forgetting to run before print (program not started).
Confusing step (into) with next (over) — wastes time exploring stdlib.
Using valgrind on a release-optimised binary (works, but reports are less precise).

Harder version. Add a deliberate buffer overflow at arr[5] and detect with ASan.

9. Written Exam Focus¶

9.1 Short Answers¶

Q. Compile flag for debug? A. -g (and usually -O0 so variables aren't optimised away).

Q. Difference between step and next. A. step enters function calls; next executes them as a single unit.

Q. What does valgrind --leak-check=full do? A. Runs the program in a memory-checking VM and reports leaks (definitely / indirectly / possibly lost / still reachable).

Q. What is a watchpoint? A. A breakpoint that triggers when an expression's value changes.

Q. What is AddressSanitizer? A. Compiler instrumentation (-fsanitize=address) detecting buffer overflows, use-after-free, etc., at runtime with ~2× overhead.

9.2 Medium Answers¶

Q. (8 marks) Walk through using gdb to find a NULL-pointer bug.

A. Compile with g++ -O0 -g. Run gdb ./prog. run → program crashes. backtrace shows the line. frame N to switch to that frame. print ptr shows it is NULL. list shows the surrounding source. Insert a fix (e.g. if (ptr) ...), recompile, retest. Use watch ptr if you suspect another path nullifies it. Common pitfall: forgetting -g so you only see addresses.

Q. (5 marks) Compare valgrind and AddressSanitizer.

A. Both detect memory errors (overflow, UAF, leaks). valgrind instruments the existing binary — no rebuild, ~30× slower. ASan compiles instrumentation in (-fsanitize=address), ~2× slower, better stack traces, but requires rebuild. valgrind when you can't recompile; ASan during development.

9.3 Long Answer (12 marks)¶

Q. Diagnose debug04.cpp step by step.

A. Introduction. debug04.cpp writes 23 into arr[count] while count < 5, but never increments count — infinite loop with a stale write to arr[0].

g++ -O0 -g debug04.cpp -o d4
gdb ./d4; run — hangs.
Ctrl-C interrupts. info locals shows count = 0, arr = {23, 2, 3, 4, 5, 6} — only arr[0] modified.
list reveals the body — the missing ++count;.
Fix, recompile.
Re-run; loop terminates.

Real HPC/CFD link. CFD time-march loops with a missing iterator update waste thousands of CPU-hours before someone notices.

Conclusion. info locals after Ctrl-C is the first move on hung programs.

9.4 Output Prediction¶

int a[5]={1,2,3,4,5}; int s=0;
for (int i=0;i<=5;++i) s+=a[i];
std::cout<<s;

→ Reads a[5] (out-of-bounds) — undefined; commonly prints 15 plus a junk integer (e.g. 32782). ASan would print "stack-buffer-overflow".

9.5 Comparison¶

step vs next — see 9.1. watch vs break — break at a line, watch on a value.

	valgrind	ASan
Recompile?	no	yes
Slowdown	~30×	~2×
Detects leaks	yes	yes (`-fsanitize=leak`)
Stack info	ok	excellent

9.6 Templates¶

Debug template: "compile with -g; gdb; break; run; print; step; fix; recompile; retest."

Valgrind template: "compile normally; valgrind --leak-check=full ./prog; read 'definitely lost' lines and trace back."

9.7 Marking Scheme — "Find off-by-one with gdb" (5 marks)¶

1 compile -g.
1 break on the loop.
1 print i and numbers[i] per iteration.
1 detect i==5 reads OOB.
1 propose fix i<5.

10. Very Hard Questions¶

Beginner

Compile flag for debug. → -g.
Start gdb. → gdb ./prog.
Set breakpoint. → break 10.
Run. → run.
Print var. → print x.

Intermediate

Step into vs over. → step / next.
Print 5-element array. → p *arr@5.
Run until end of function. → finish.
Show stack. → backtrace.
Detect leaks. → valgrind --leak-check=full.

Hard

Why does gdb show <optimized out>? → compiled with -O2 or higher.
Pause when i becomes 100. → watch i, then condition N i==100.
Inspect a CPU register. → info registers / p $rax.
Set variable from gdb. → set var x = 5.
Attach to a running process. → gdb -p <pid>.

Very Hard

Reverse-debug? → record full; reverse-step (gdb feature).
Debug an optimised release build? → -O2 -g + -fno-omit-frame-pointer.
Debug an MPI rank crash? → run ranks under gdb (xterm trick) or attach to the PID.

Deep Integration

Sanitizers in CI. → build with -fsanitize=address,undefined; tests fail on any error.
Why are leaks fatal in long CFD runs? → memory grows per iteration → OOM kills the job.

Coding/Command

One-liner compile + run with ASan. → g++ -O1 -g -fsanitize=address prog.cpp && ./a.out.
gdb session for debug03. → see 6.3.

Debugging

"core dumped" but no core file. → ulimit -c unlimited.
valgrind: "Conditional jump depends on uninitialised value(s)". → uninit read; rerun with --track-origins=yes.

Long Written

(250 words) Demonstrate gdb on debug05 — step-by-step trace.

11. Debugging and Mistake Analysis¶

Mistake	Why wrong	Correct	Explanation
Compile w/o `-g`	gdb shows no source	add `-g`	DWARF info
Compile `-O3` for debug	variables gone	`-O0 -g`	optimisations remove vars
Forgot `run`	print errors	run first	program must start
Ctrl-D in gdb	exits gdb	use `quit`	habit
Confusing step/next	wastes time in libc	`next` for unknown calls	choose carefully
valgrind on stripped binary	bad backtrace	rebuild with `-g`	symbols required
Forgot free/delete	leak	RAII / smart ptrs	modern C++
`delete arr` for array	UB	`delete[] arr`	array form
`if (a = b)` typo	always assigns	`==`	classic
Reading core without exe	gdb confused	`gdb ./exe core`	both needed

12. Mini Project for Mastery¶

Goal: Take task01.cpp (Mapper2D) and produce a fixed, sanitizer-clean version with unit tests.

# 1. Add unit tests in test.cpp
# 2. Build with sanitizers
g++ -O1 -g -fsanitize=address,undefined -Wall task01.cpp test.cpp -o map_test
# 3. Run
./map_test
# 4. valgrind for leaks
valgrind --leak-check=full ./map_test
# 5. Profile
g++ -O2 -pg task01.cpp -o map_prof; ./map_prof; gprof map_prof gmon.out > prof.txt

test.cpp:

#include <cassert>
extern int xyToPos(int,int,int,int);
extern int posToX (int,int,int);
extern int posToY (int,int,int);
int main(){
    for (int y=0; y<5; ++y)
        for (int x=0; x<5; ++x) {
            int p = xyToPos(x,y,5,5);
            assert(posToX(p,5,5)==x);
            assert(posToY(p,5,5)==y);
        }
}

Connection to exam: building with sanitizers, asserting invariants, gdb when a test fails.

13. Final Chapter Cheat Sheet¶

Item	Memorise
Build for debug	`g++ -O0 -g …`
Start	`gdb ./prog`
Breakpoint	`break 10` / `break func`
Run	`run [args]`
Step	`step` / `s` (into)
Next	`next` / `n` (over)
Continue	`c`
Print	`p var` / `p *arr@5`
Stack	`bt` / `frame N`
Watch	`watch var`
Quit	`q`
Memory	`valgrind --leak-check=full --track-origins=yes ./p`
ASan	`-fsanitize=address,undefined`
TSan	`-fsanitize=thread`
Core	`ulimit -c unlimited; gdb ./p core`
MPI	`mpirun -np N xterm -e gdb --args ./p`
Bisection	bug among N commits → $\lceil \log_2 N \rceil$ tests
Trap	`<optimized out>` after `-O2`
Top phrase	"Compile -g, run gdb, break + run + print + step + fix; valgrind / ASan find memory bugs that gdb alone misses."

14. Mock Exam — Four Levels¶

Level 1 — Basic (definitions & syntax)¶

Q1. Which two compile flags prepare a binary for gdb, and why both?

Solution: -g (embed debug symbols) and -O0 (keep variables/lines un-optimised so they're inspectable).

Q2. In gdb, how do you (a) stop at function solve, (b) start the program, (c) show the call stack?

Solution: (a) break solve, (b) run, (c) backtrace (bt).

Q3. Which tool finds a memory leak WITHOUT recompiling?

Solution: valgrind --leak-check=full ./prog.

Q4. What does p *arr@5 print?

Solution: The first 5 elements of array arr (gdb artificial-array syntax).

Q5. What signal does a NULL-pointer dereference raise?

Solution: SIGSEGV (segmentation fault).

Level 2 — Intuitive (predict / explain why)¶

Q1. You set break 12 and type print x — gdb says "No symbol x in current context". The program hasn't crashed. Most likely cause?

Solution: You forgot run — no frame exists yet (or x is out of scope at line 12).

Q2. gdb shows i = <optimized out>. Explain and fix.

Solution: Built with -O2+ — the compiler kept i in a register or eliminated it. Rebuild with -O0 -g.

Q3. valgrind reports "still reachable: 72,704 bytes" for a trivial program. Should you panic?

Solution: No — "still reachable" blocks (often C++ runtime internals) are still pointed-to at exit; the categories to fix are "definitely lost" / "indirectly lost".

Q4. Why does a use-after-free often NOT crash immediately?

Solution: Freed memory usually stays mapped in the allocator's pool; reads return stale data silently until the block is reused. That's why ASan/valgrind (which poison/track freed memory) are needed.

Q5. A program crashes only with -O3, never with -O0. Is the compiler buggy?

Solution: Almost always no — the code has undefined behaviour (OOB, uninit read, aliasing) that optimisation exposes. Run UBSan/ASan to find the UB.

Level 3 — Hard (exam level)¶

Q1. (10 marks) Given this output, name the bug class, the offending location, and the fix:

==7777== Invalid write of size 8
==7777==    at 0x401189: fill (vec.cpp:9)
==7777==    by 0x4011F2: main (vec.cpp:17)
==7777==  Address 0x4dbc090 is 0 bytes after a block of size 80 alloc'd
==7777==    at 0x483B7F3: operator new[] (...)
==7777==    by 0x401151: main (vec.cpp:15)

Solution: Heap buffer overflow (out-of-bounds write). The block is 80 B = 10 doubles; the write at vec.cpp:9 in fill lands "0 bytes after" the block ⇒ element index 10 ⇒ the loop bound uses <= (or the size is off by one). Fix the loop in fill to i < 10 (or allocate 11). Re-run valgrind to confirm zero errors. বাংলা ইঙ্গিত: "0 bytes after a block of size 80" পড়তে শেখো: ব্লকের ঠিক পরের byte-এ লেখা — মানে index == size, classic off-by-one।

Q2. (8 marks) Your simulation "worked at v1.0, broken now" across 1024 commits. How many builds must you test with git bisect, and what are the exact starting commands?

Solution: $\lceil \log_2 1024 \rceil = 10$ tests.

git bisect start
git bisect bad HEAD
git bisect good v1.0
# build+test, then mark: git bisect good|bad   (repeat ~10×)
git bisect reset

বাংলা ইঙ্গিত: bisection-এর গণিত: প্রতি টেস্টে সন্দেহ-তালিকা অর্ধেক — তাই log₂N; 1024 → 10, এক মিলিয়ন → মাত্র 20।

Q3. (8 marks) A colleague's program hangs on the cluster. You may not restart it. Give the diagnosis procedure with exact commands.

Solution: Find the PID: ps aux | grep solver (via ssh to the compute node from squeue info). Attach: gdb -p <pid>. Inspect: bt (where is it stuck), info locals (loop counters), info threads + thread apply all bt for multithreaded deadlock. Leave it running: detach; quit. বাংলা ইঙ্গিত: hang মানেই kill নয় — gdb -p দিয়ে জীবন্ত প্রসেসে ঢুকে bt দেখা যায়, আর detach করলে জব চলতেই থাকে।

Q4. (10 marks) Choose the right tool (gdb / valgrind / ASan / TSan) for each failure, one-line justification: (a) crash at unknown line in a production binary you cannot rebuild; (b) results differ run-to-run with OpenMP; (c) RAM grows 1 MB per timestep; (d) reproducible segfault in your own dev build.

Solution: (a) gdb + core dump — no rebuild needed, bt gives the location. (b) TSan (-fsanitize=thread) — nondeterminism smells like a data race. (c) valgrind --leak-check=full — steady growth = per-iteration leak. (d) ASan first (fast, precise trace), then gdb to step the failing path. বাংলা ইঙ্গিত: উপসর্গ→টুল ম্যাপিং মুখস্থ: rebuild-নিষেধ = valgrind/gdb, race = TSan, leak = valgrind, নিজের বিল্ড = ASan।

Q5. (10 marks) Write a tiny program WITH a use-after-free, the ASan command to expose it, and the first line of the expected report.

Solution:

#include <iostream>
int main(){
    int* p = new int[4];
    delete[] p;
    std::cout << p[1] << "\n";   // use after free
}

g++ -O1 -g -fsanitize=address uaf.cpp -o uaf && ./uaf First line: ==<pid>==ERROR: AddressSanitizer: heap-use-after-free on address 0x... বাংলা ইঙ্গিত: ASan-রিপোর্টের প্রথম লাইনেই error-class-এর নাম — পরীক্ষায় ঠিক ওই শব্দবন্ধ ("heap-use-after-free") লিখতে পারাটাই নম্বর।

Level 4 — Beyond the lecture (transfer + coding)¶

Q1. An MPI run with 64 ranks crashes on exactly one rank with SIGSEGV after ~2 h. Design a capture strategy that doesn't slow the other 63 ranks.

Solution: Post-mortem core dumps: put ulimit -c unlimited in the jobscript, rerun, then gdb ./solver core.<pid> → bt full on the one core file. Zero runtime cost for healthy ranks. (Alternative: conditionally wrap one suspect rank in gdb via its rank env var.) বাংলা ইঙ্গিত: দীর্ঘ জবে live-debug নয়, post-mortem — core dump-ই সবচেয়ে সস্তা সাক্ষী; শুধু ulimit আগে খুলে রাখতে হয়।

Q2. Write a Makefile target debugcheck (Ch 12 transfer) that builds with ASan+UBSan and runs the test suite, failing the build on any sanitizer report.

Solution:

debugcheck: CXXFLAGS += -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer
debugcheck: clean prog
    ASAN_OPTIONS=halt_on_error=1 ./prog --selftest
.PHONY: debugcheck

Target-specific variables add sanitizer flags only here; halt_on_error=1 turns the first report into a nonzero exit, failing make. বাংলা ইঙ্গিত: "target-specific variable" (target: VAR += …) — Makefile-এর কম-পরিচিত কিন্তু পরীক্ষা-জেতানো ফিচার।

Q3. Your solver gives slightly different residuals every run, single-threaded, same input. gdb shows nothing. Name the two most likely classes of cause and the detection tool for each.

Solution: (1) Uninitialised memory feeding the computation → valgrind --track-origins=yes; (2) out-of-bounds read that doesn't crash → ASan. (Multithreaded nondeterminism would be a race → TSan, but single-thread excludes it.) বাংলা ইঙ্গিত: "এক input-এ ভিন্ন ফল" + single thread = প্রায় নিশ্চিত uninitialised/OOB read — crash করে না বলেই sanitizer লাগে।

Q4. The fixed Mapper2D test passes but runs 100× slower after someone's "fix": posToX now calls printf on every call. Give the gprof workflow that catches such a regression and the expected top line.

Solution:

g++ -O2 -pg task01.cpp test.cpp -o map_prof
./map_prof
gprof map_prof gmon.out | head

Flat-profile top line: posToX (or vfprintf) with a huge call count consuming ~all runtime. Remove I/O from the hot path. বাংলা ইঙ্গিত: correctness পাশ মানেই শেষ নয় — hot loop-এ I/O ঢোকানো classic "fix that breaks", আর profiler-ই তার একমাত্র চোখ।

End of Chapter 13. er (line 42) that is never freed; after 100 steps, 4 MB leaked, growing linearly → OOM in long runs. Buggy pattern:

void time_step() {
    double *scratch = (double*)malloc(5000 * sizeof(double)); // solver.cpp:42
    // ... use scratch ...
    // missing free(scratch);
}

Fix 1 (C): free(scratch); at the end of every path out of the function. Fix 2 (modern C++, better): std::vector<double> scratch(5000); — RAII frees automatically; or allocate once outside the time loop and reuse.

বাংলা ইঙ্গিত: "100 blocks" সংখ্যাটাই সূত্র — ঠিক time-step সংখ্যার সমান, মানে leak টা loop-এর ভেতরে; backtrace-এর malloc লাইনটা (42) উত্তরে উদ্ধৃত করো।

Q3.4 A field value rho mysteriously becomes negative somewhere in a long run. Write a complete gdb session that stops at the exact moment rho turns negative, using a watchpoint with a condition.

Solution:

$ g++ -O0 -g solver.cpp -o solver
$ gdb ./solver
(gdb) break main            # get into a frame where rho is in scope
(gdb) run in.dat
(gdb) watch rho             # hardware watchpoint: stop on ANY change
Hardware watchpoint 2: rho
(gdb) condition 2 rho < 0   # ...but only when the new value is negative
(gdb) continue
Hardware watchpoint 2: rho
Old value = 1.184
New value = -0.0031
update_density (cell=4711) at solver.cpp:217
(gdb) backtrace             # who called the corrupting function
(gdb) print cell            # which cell — feed back into CFD analysis

The watchpoint fires on every write to rho; the condition filters stops to the first negative write; backtrace then reveals the call chain and line (217) that produced it.

বাংলা ইঙ্গিত: watch + condition-এর যুগলবন্দিই এখানে আসল উত্তর — শুধু watch দিলে হাজারবার থামবে, শুধু break দিলে ভুল জায়গায় থামবে।

Q3.5 Choose the right tool (with the exact diagnostic phrase you expect to read) for each failure: (a) RAM grows ~1 MB per iteration over hours; (b) instant segmentation fault at startup; (c) multithreaded results differ run to run; (d) suspected silent corruption of a stack array.

Solution:

(a) Leak → valgrind --leak-check=full ./prog; expect "N bytes in M blocks are definitely lost"; backtrace names the malloc line. (LeakSanitizer is the fast alternative.)
(b) Crash location → gdb ./prog, run, then backtrace; expect "Program received signal SIGSEGV" with file:line (typical cause: NULL dereference).
(c) Data race → recompile -fsanitize=thread; expect "WARNING: ThreadSanitizer: data race" with both conflicting stack traces (valgrind's helgrind is the no-recompile option).
(d) Stack out-of-bounds → ASan, -fsanitize=address; expect "stack-buffer-overflow". valgrind would stay silent — it cannot see stack-array bounds.

বাংলা ইঙ্গিত: (d)-টাই বাছাই-প্রশ্নের কাঁটা: stack array মানে valgrind অন্ধ, ASan-ই একমাত্র উত্তর — এই asymmetry-টা না লিখলে প্রশ্নটার মূল পয়েন্টই বাদ।

Level 4 — Beyond the lecture (transfer questions)¶

Q4.1 Write a CI-style bash script check.sh that builds the solver with ASan+UBSan, runs the test binary, and exits non-zero (failing the pipeline) on any sanitizer finding or test failure.

Solution:

#!/bin/bash
set -euo pipefail            # any failing command aborts the script

g++ -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer \
    solver.cpp test.cpp -o solver_test

# halt_on_error: first ASan finding terminates with non-zero exit;
# UBSan likewise made fatal so it cannot just print-and-continue.
export ASAN_OPTIONS=halt_on_error=1:detect_leaks=1
export UBSAN_OPTIONS=halt_on_error=1

./solver_test
echo "ALL CHECKS PASSED"

Mechanism: set -e propagates the test binary's exit code; ASan aborts with non-zero status on the first error because of halt_on_error=1, and detect_leaks=1 also fails the run on leaks at exit. In CI (e.g. GitLab CI), a non-zero exit of check.sh marks the job red.

বাংলা ইঙ্গিত: শুধু compile+run লিখলে অর্ধেক — exit code-এর মাধ্যমে pipeline fail করানোর ব্যবস্থা (set -e, halt_on_error) দেখানোই প্রশ্নের আসল দাবি।

Q4.2 A CFD residual becomes NaN at iteration 2317 of a long run. Show how to make the creation of the NaN trap into the debugger: code snippet plus gdb session.

Solution: Enable floating-point exceptions so the first invalid operation raises SIGFPE instead of silently producing NaN:

#include <cfenv>
// at the top of main():
feenableexcept(FE_INVALID | FE_DIVBYZERO | FE_OVERFLOW);  // glibc

$ g++ -O0 -g solver.cpp -o solver
$ gdb ./solver
(gdb) run in.dat
...iterations run...
Program received signal SIGFPE, Arithmetic exception.
compute_flux (f=0x55..., i=8812) at flux.cpp:131
131        q = mdot / rho;        ◄── rho == 0 here: division by zero
(gdb) print rho
$1 = 0
(gdb) backtrace                   ◄── full call chain to the bad cell
(gdb) frame 1                     ◄── inspect the caller's state

Without the trap, the NaN propagates silently through the whole field and only shows up in the residual much later; with feenableexcept + gdb you stop at the first invalid operation — the true origin. (Alternative for Fortran-style workflows: -ffpe-trap=invalid,zero,overflow in gfortran.)

বাংলা ইঙ্গিত: NaN নিজে crash করায় না — সেটাই সমস্যা; feenableexcept দিয়ে NaN-জন্মকে signal-এ বদলে দিলেই gdb-র সব অস্ত্র (bt, frame, print) কাজে লেগে যায়।

Q4.3 Write a Makefile with three targets — release, debug, asan — that build solver.cpp into differently named binaries with appropriate flags, and explain in two sentences why debug and release must be separate binaries.

Solution:

CXX      = g++
SRC      = solver.cpp
WARN     = -Wall -Wextra

release: $(SRC)
    $(CXX) -O3 -march=native -DNDEBUG $(WARN) $(SRC) -o solver

debug: $(SRC)
    $(CXX) -O0 -g $(WARN) $(SRC) -o solver_dbg

asan: $(SRC)
    $(CXX) -O1 -g -fsanitize=address,undefined \
           -fno-omit-frame-pointer $(WARN) $(SRC) -o solver_asan

clean:
    rm -f solver solver_dbg solver_asan

Usage: make debug && gdb ./solver_dbg. Why separate: debug needs -O0 -g so every variable and line is faithful to the source, while production needs -O3 -DNDEBUG for speed (and NDEBUG removes asserts) — one binary cannot serve both, and sanitizer instrumentation (~2× slower, different memory layout) must never ship into the cluster production run.

বাংলা ইঙ্গিত: Makefile-এ recipe-লাইনগুলো TAB দিয়ে শুরু করতেই হবে (space নয়) — লিখিত পরীক্ষাতেও এটা উল্লেখ করা নিরাপদ, কারণ এটাই make-এর কুখ্যাত syntax-ফাঁদ।

Q4.4 A 16-rank MPI CFD job dies with SIGSEGV — but only on rank 12; the other 15 ranks are fine. Give the complete post-mortem workflow on the cluster (commands), and explain why a single rank can crash while identical code runs on all ranks.

Solution:

# 1. in the jobscript, before mpirun: allow core dumps
ulimit -c unlimited
mpirun -n 16 ./solver_dbg in.dat        # use the -O0 -g build

# 2. after the crash: find rank 12's core file in the rundir
ls core*                                 # e.g. core.184223  (one per crashed rank)

# 3. load executable + core into gdb (no rerun needed)
gdb ./solver_dbg core.184223
(gdb) backtrace                          # crash line + full call chain
(gdb) frame 2
(gdb) print i                            # the offending index
(gdb) print rank                         # confirm: 12

# live alternative on an interactive allocation:
mpirun -n 16 xterm -e gdb --args ./solver_dbg in.dat

Why only rank 12: the code is identical (SPMD) but the data is not — domain decomposition gives every rank a different sub-domain. Rank 12's partition contains the special situation (e.g. a boundary patch, an empty neighbour list, a cell count that makes an off-by-one index reach past its halo array). Bug class: data-dependent out-of-bounds — exactly why MPI debugging starts from the crashing rank's core file, not from rank 0.

বাংলা ইঙ্গিত: "সব rank একই code, তবু একটাই crash" — উত্তরটা সবসময় data-র অসমতা: domain decomposition-এ প্রতিটা rank-এর subdomain আলাদা, আর bug-টা data-নির্ভর।

End of Chapter 13.

Chapter 13: Debugging — gdb, valgrind, sanitizers¶

1. Chapter Overview¶

2. Basics from Zero¶

3. Hard English Made Easy¶

4. Deep Theory Explanation¶

4.1 Compile for debugging¶

4.2 gdb command catalogue¶

4.3 valgrind essentials¶

4.4 Sanitizers (modern preferred)¶

4.5 The 6 lecture bugs¶

4.6 Diagrams from V11¶

4.7 The mathematics of bug hunting — bisection¶

4.8 Memory-error taxonomy — which tool catches what¶

4.9 Annotated gdb session — catching an off-by-one live¶

5. Command / Syntax / Code Breakdown¶

g++ -g -O0 file.cpp -o exe¶

gdb ./exe¶

(gdb) break 8¶

(gdb) break add¶

(gdb) run¶

(gdb) next / step¶

(gdb) print x / (gdb) p *arr@5¶

(gdb) watch i¶

(gdb) backtrace¶

valgrind --leak-check=full ./exe¶

-fsanitize=address¶

6. Mandatory Practical Examples¶

Example 6.1 — Off-by-one (debug01.cpp)¶

Example 6.2 — Trace add function (debug02.cpp)¶

Example 6.3 — NULL deref (debug03.cpp)¶

Example 6.4 — Infinite loop (debug04.cpp)¶

Example 6.5 — Logical bug (debug05.cpp)¶

Example 6.6 — Memory leak (debug06.cpp)¶

Example 6.7 — E11 Mapper2D (task01.cpp)¶

7. Real HPC/CFD Workflow¶

8. Exercises and Solutions¶

9. Written Exam Focus¶

9.1 Short Answers¶

9.2 Medium Answers¶

9.3 Long Answer (12 marks)¶

9.4 Output Prediction¶

9.5 Comparison¶

9.6 Templates¶

9.7 Marking Scheme — "Find off-by-one with gdb" (5 marks)¶

10. Very Hard Questions¶

11. Debugging and Mistake Analysis¶

12. Mini Project for Mastery¶

13. Final Chapter Cheat Sheet¶

14. Mock Exam — Four Levels¶

Level 1 — Basic (definitions & syntax)¶

Level 2 — Intuitive (predict / explain why)¶

Level 3 — Hard (exam level)¶

Level 4 — Beyond the lecture (transfer + coding)¶

Level 4 — Beyond the lecture (transfer questions)¶

`g++ -g -O0 file.cpp -o exe`¶

`gdb ./exe`¶

`(gdb) break 8`¶

`(gdb) break add`¶

`(gdb) run`¶

`(gdb) next` / `step`¶

`(gdb) print x` / `(gdb) p *arr@5`¶

`(gdb) watch i`¶

`(gdb) backtrace`¶

`valgrind --leak-check=full ./exe`¶

`-fsanitize=address`¶