Chapter 7: Linux Text Tools — find, grep, awk, sed, diff, column¶

Source slides: V05_Linux_tools.pdf (full lecture). Exercise: E05_Linux_tools.pdf (+ solutions). Data files: V05_examples/ (grep/employee_database.txt, awk/script01.awk, sed/script02.sed, find/, column/column.txt, diff/diff1.txt diff2.txt, numbers.txt, test.sh).

1. Chapter Overview¶

This chapter covers the core text-processing toolbox of Linux — the tools you'll use 100 times a day on any HPC cluster. They come in five families:

Family	Tools	Job
Filename search	`find`, `locate`	Find files by attributes
Text search	`grep`, `egrep`, `fgrep`	Find lines
Stream editing	`sed`	Find-and-replace per stream
Field processing	`awk`	Column / record arithmetic
Compare	`diff`, `cmp`, `comm`	Compare files
Tabular formatting	`column`	Pretty-print delimited text

Why it matters in HPC/CFD: a CFD log is a 2 GB stream of numbers. awk summarises it; sed rewrites parameters; grep finds the line where the residual blew up; diff compares two simulation outputs; find tags every .cpp and chmod's them.

What the examiner asks (every exam):

"Difference between grep, awk, and sed."
"Predict output of awk -F, '{print $1,$3}' file."
"Write a sed command to replace foo with bar in place."
"Use find to locate .cpp files and chmod them."
"Pipeline: list employees in the Engineering dept earning > 60 000."

What you must master for top grade:

The mental model grep selects lines, awk selects fields/lines + computes, sed rewrites streams.
find syntax (-name, -type, -size, -mtime, -exec, -regex).
awk programs: BEGIN { } /pattern/ { action } END { }, FS/OFS, $1..$NF, NR, NF.
sed substitution: s/pat/repl/flags, ranges, -i, -E, -n + p.
The lecture's E05 solutions verbatim.
column -t -s ":" for tabular output.

2. Basics from Zero¶

Linux text processing follows the Unix philosophy: small tools, plain text, joined by pipes. Each tool reads stdin, writes stdout, and does one thing.

grep prints lines that match a pattern — selection.
sed prints lines after applying edits — transformation.
awk treats each line as a record split into fields, runs a small program — analysis.
find walks a directory tree and applies tests (-name, -size) and actions (-exec).
diff prints the differences between two files.
column prettifies a delimited file into a table.

A real CFD pipeline:

grep -E 'residual' run.log \              # selection
  | awk '{print $2, $5}' \                # extraction
  | sed 's/Iter//' \                       # cleaning
  | column -t \                            # formatting
  > clean_residuals.dat                    # save

Real-life analogy.

grep = highlighter that keeps only "matching" sentences.
sed = find-and-replace robot that rewrites every line that passes by.
awk = mini-spreadsheet at the command line.
find = "Search across the file tree" with predicates.
diff = side-by-side comparison.
column = neat-table formatter.

What if you misunderstand? You use grep for arithmetic ("salary > 60 000") — but that's awk's job. Or you use sed to find files — but that's find. Picking the right tool is half the answer.

বাংলায়: এই অধ্যায়ের চারটে অস্ত্র মনে রাখো এক লাইনে: find ফাইল খোঁজে, grep লাইন বাছে, sed লেখা বদলায়, awk কলাম নিয়ে হিসাব করে। পরীক্ষার বেশিরভাগ প্রশ্নই এই চারটার সঠিক জুটি বানানো নিয়ে।

3. Hard English Made Easy¶

Hard Term	Simple English	বাংলা	Example
Stream	Flow of bytes through stdin/stdout	বাইটের ধারা	`cat f \\| sed s/a/b/`
Field	Piece of a line separated by FS	লাইনের একটি অংশ	`$1`, `$2` in awk
Record	One line in awk	এক লাইন	each `\n`
FS / OFS	Field / Output Field Separator	ফিল্ড আলাদাকারী	`,` `\t`
RS / ORS	Record / Output Record Separator	রেকর্ড আলাদাকারী	`\n`
Predicate (find)	Test like `-name`, `-type`	শর্ত চিহ্ন	`-name "*.cpp"`
In-place edit (sed)	Modify file directly	ফাইল সরাসরি বদলানো	`sed -i ...`
Line range	Subset of lines to act on	লাইনের সীমা	`5,10s/...`
Hold space (sed)	Auxiliary buffer	সহকারী বাফার	rare in basic use
Pattern space (sed)	Current line buffer	বর্তমান লাইন	default
Delimiter	Separator char	পৃথককারী চিহ্ন	`,` `:`
Pretty-print	Format readably	পরিপাটি প্রিন্ট	column
Diff hunk	Block of changes	পরিবর্তনের ব্লক	`@@` lines

4. Deep Theory Explanation¶

4.1 `find`¶

find <path> <expression>

Common predicates:

Predicate	Meaning
`-name "*.cpp"`	name match (glob)
`-iname "*.CPP"`	case-insensitive
`-regex "./.*/[Ss]ource.+\.cpp$"`	regex on full path
`-type f / d / l`	file / dir / symlink
`-size +1M`	larger than
`-mtime -1`	modified < 1 day ago
`-mmin +60`	modified > 60 min ago
`-user alice`	owned by
`-perm 644` / `-perm -u+x`	permission
`-empty`	size 0
`-maxdepth N`, `-mindepth`	tree depth
`-not`, `!`, `-or`, `-and` (default)	logic

Actions (terminate the predicate):

Action	Meaning
`-print`	print path (default)
`-print0`	NUL-terminate (safe with weird names)
`-delete`	delete
`-exec cmd {} \;`	run cmd per file
`-exec cmd {} +`	run cmd in batches (faster)
`-ok cmd {} \;`	like exec, ask first

Examples:

find . -type f -name "*.cpp" -exec wc -l {} +
find . -size +100M -mtime +30
find . -regex './.*/[Ss]ource.+\.cpp$' -exec chmod +w {} \;
find . -name "*.log" -delete

   find expression evaluation (left → right, short-circuit AND)
   ┌──────────┐    ┌──────────────┐    ┌──────────────┐    ┌─────────────────┐
   │ start at │───►│ -type f      │───►│ -name "*.o"  │───►│ -exec rm {} \;  │
   │ each path│    │ test: file?  │    │ test: match? │    │ ACTION (runs    │
   └──────────┘    └──────┬───────┘    └──────┬───────┘    │ only if all     │
                          │ false             │ false      │ tests passed)   │
                          ▼                   ▼            └─────────────────┘
                     skip path            skip path
   Implicit -and between predicates; -or needs explicit grouping \( ... \)

বাংলায়: find-এর expression গুলো বাঁ থেকে ডানে এক-একটা শর্ত (predicate) হিসেবে চলে, আর মাঝে অদৃশ্য AND থাকে — কোনো শর্ত fail করলেই পরেরগুলো আর দেখা হয় না (short-circuit)। -exec আসলে শর্ত নয়, action — সব শর্ত পাশ করলে তবেই চলে। তাই -delete বা -exec সবসময় শেষে লেখো; আগে লিখলে শর্ত পরীক্ষার আগেই কাজ হয়ে যাবে।

4.2 `grep`¶

grep [options] PATTERN [file...]

Useful options:

Flag	Meaning
`-E`	ERE (alias `egrep`)
`-F`	fixed string
`-P`	PCRE
`-i`	case-insensitive
`-v`	invert
`-n`	line number
`-c`	count
`-l`	filenames only
`-L`	files without match
`-r` / `-R`	recursive
`-w`	whole word
`-x`	whole line
`-A N` / `-B N` / `-C N`	context after / before / both
`-o`	only matched part
`-q`	quiet (exit status only)
`--include="*.cpp"` / `--exclude`	filter files

বাংলায়: grep-এর core idea: প্রতি লাইনে pattern খোঁজা, মিললে লাইনটা প্রিন্ট। -v উল্টে দেয় (না-মেলা লাইন), -l শুধু ফাইলের নাম, -c শুধু সংখ্যা, -o শুধু মিলে-যাওয়া অংশটুকু। পরীক্ষায় flag-combination প্রশ্ন আসে — যেমন "কোন ফাইলগুলোতে ERROR আছে শুধু নামগুলো দাও" মানেই grep -l।

4.3 `sed`¶

sed [options] 'commands' [file...]

Important options: -E (ERE), -i (in-place; sed -i.bak keeps backup), -n (no auto-print, used with p).

Common commands:

Command	Meaning
`s/pat/repl/flags`	substitute (flags: `g` all, `i` case-i, `p` print)
`d`	delete line
`p`	print
`q`	quit
`y/abc/xyz/`	transliterate
`r file`	read & insert
`5,10s/.../...`	line range
`/pat/d`	delete matching
`1!d`	delete all except line 1

Substitution can use back-references \1..\9 and the matched text &.

sed 's/foo/bar/'           # first per line
sed 's/foo/bar/g'          # all per line
sed -i.bak 's/foo/bar/g' f # in-place with backup
sed -n '/ERROR/p' run.log  # print only ERROR lines
sed -E 's/Re *= *[0-9]+/Re = 5000/'
sed '5,10s/^/# /'          # comment lines 5..10

The sed cycle (pattern-space model). For every input line sed: (1) copies the line into the pattern space, (2) applies every command in order, (3) auto-prints the pattern space (unless -n), (4) clears it and reads the next line.

        ┌────────────────────────────────────────────────────┐
        │                    sed CYCLE                        │
        │  read line i ──► PATTERN SPACE ──► apply commands   │
        │       ▲           (one line)        s/d/p/y ...     │
        │       │                                  │          │
        │       │                                  ▼          │
        │  next line  ◄── clear ◄── auto-print (skip if -n)   │
        └────────────────────────────────────────────────────┘
        HOLD SPACE (side buffer): h/H copy in, g/G copy back — rarely
        needed in the course, but name it for top marks.

বাংলায়: sed মানে stream editor — পুরো ফাইল মেমরিতে নেয় না, এক লাইন করে pattern space-এ এনে কমান্ড চালিয়ে প্রিন্ট করে দেয়। -n দিলে auto-print বন্ধ — তখন p flag/command দিয়ে বেছে বেছে প্রিন্ট করা যায়; sed -n '/ERROR/p' তাই grep-এর মতো কাজ করে। এই cycle-টা বুঝলে sed-এর সব আচরণ অনুমান করা যায়।

4.4 `awk`¶

awk is a small programming language. A program is a sequence of pattern { action } rules; for each line, awk runs every rule whose pattern matches.

awk 'BEGIN { ... }
     /pattern/ { ... }
     END { ... }' file

Built-in variables:

Var	Meaning
`$0`	whole line
`$1, $2, …`	fields
`NR`	current record number (line)
`NF`	number of fields on this line
`FILENAME`	current input file
`FS` / `OFS`	input / output field separator (default whitespace)
`RS` / `ORS`	input / output record separator (default `\n`)
`FNR`	record number per file

Common idioms:

awk '{print $1}' file                   # 1st field
awk -F, '{print $1,$3}' f.csv           # CSV columns 1+3
awk '$3 > 1000' f                       # rows where col3 > 1000
awk 'NR==1 || NR==5'                    # specific lines
awk 'NR>1' f                             # skip header
awk '{s+=$2} END{print s}'               # sum col 2
awk '{a[$1]++} END{for (k in a) print k,a[k]}'   # word count
awk 'BEGIN{FS=",";OFS="\t"} {print $1,$2}'       # CSV → TSV
awk '/pattern/ {print NR":"$0}'         # like grep -n
awk 'length($0)>80'                      # long lines
awk '{printf "%-10s %8.3f\n",$1,$2}'      # formatted

The awk model, formally. Input is split into records (default: lines) and each record into fields $1..\$NF$ by the separator FS. For each record, awk evaluates every pattern { action } rule; BEGIN/END run before/after the data.

Worked numeric example 1 — sum and mean of column 2. Input times.dat:

step1 0.42
step2 0.38
step3 0.45
step4 0.51

awk '{s+=$2} END{print "sum =", s, " mean =", s/NR}' times.dat

Trace: s accumulates 0.42 → 0.80 → 1.25 → 1.76; NR ends at 4. Output: sum = 1.76 mean = 0.44.

Worked numeric example 2 — maximum with position.

awk 'NR==1 || $2>max {max=$2; at=NR} END{print "max =", max, "at line", at}' times.dat

Trace: line1 sets max=0.42; line3 (0.45>0.42) updates; line4 (0.51>0.45) updates. Output: max = 0.51 at line 4.

Worked numeric example 3 — conditional count (how many steps slower than 0.44?).

awk '$2>0.44 {n++} END{print n+0}' times.dat

Lines 3 (0.45) and 4 (0.51) pass → output 2. (n+0 prints 0 instead of an empty string when nothing matches.)

বাংলায়: awk-এর তিনটা মন্ত্র: (১) প্রতিটা লাইন আপনাআপনি field-এ ভাগ হয় ($1, $2 …), (২) pattern { action } — pattern মিললেই action চলে, (৩) END ব্লক সব লাইন শেষ হলে চলে — তাই sum/mean/max সবসময় END-এ প্রিন্ট করতে হয়। NR মানে কত নম্বর লাইন, NF মানে এই লাইনে কয়টা field — এ দুটো গুলিয়ো না।

4.5 `diff`¶

diff [options] file1 file2

Options: -u unified, -c context, -y side-by-side, -r recursive, -q brief, -i ignore case, -w ignore whitespace.

Hunk format @@ -l,c +l,c @@. Patches: diff -u a b > a.patch && patch -p0 < a.patch.

4.6 `column`¶

Pretty-print into table.

column -t file                       # auto-detect whitespace
column -t -s : /etc/passwd           # custom separator
column -t -s , file.csv > out.txt

4.7 Diagram explanations from V05¶

"Pipeline diagram": cat → grep → awk → sort → uniq -c → sort -nr. Write: "Each tool reads stdin and writes stdout; pipes connect them; this composes complex transforms from simple primitives."
"awk records and fields": a CSV row split into $1 $2 $3 … with NF and NR.
"sed two-buffer model": pattern space (current line being processed) and hold space (auxiliary).

Pipeline data-flow, stage by stage. What the data looks like as it passes through a real pipeline:

 data.csv: "# comment", "Alice,Eng,72000", "Bob,Sales,51000", "Carol,Eng,64000"
      │
      ▼
 grep -v '^#' data.csv          │ kills comment lines
      │  "Alice,Eng,72000" / "Bob,Sales,51000" / "Carol,Eng,64000"
      ▼
 awk -F, '$2=="Eng"{s+=$3; n++} END{print s/n}'
      │  selects Eng rows: 72000, 64000 → s=136000, n=2
      ▼
 output: 68000                  │ mean Engineering salary

বাংলায়: pipe-এর দর্শন: প্রতিটা tool ছোট একটা কাজ করে, আর | একটার output পরেরটার input বানায়। পরীক্ষায় "এই pipeline-এর output কী?" এলে প্রতিটা stage-এর পরে ডেটা কেমন দাঁড়ায় সেটা ধাপে ধাপে লিখে দেখাও — উপরের ছকের মতো।

5. Command / Syntax / Code Breakdown¶

`find . -name "*.txt" -type f`¶

Purpose: locate text files. Exam tip: quote the pattern so the shell doesn't expand it.

`find ... -exec chmod +w {} \;`¶

Purpose: apply a command per file. {} placeholder, \; ends the exec clause.

`grep -E "(foo|bar)" file`¶

Purpose: ERE alternation. Without -E, you'd write \|.

`grep -nA 2 -B 1 ERROR run.log`¶

Purpose: show line numbers and context.

`sed -E -i.bak 's/^Re = [0-9]+/Re = 5000/' file`¶

Purpose: in-place edit with backup (.bak), ERE.

`awk -F, 'NR>1 && $4=="Engineering" && $6>60000 {print $1,$2,$3}' employee.csv`¶

Purpose: salary filter — typical exam question.

`diff -u a b`¶

Purpose: unified diff.

`column -t -s ":" /etc/passwd > new_file.txt`¶

Purpose: pretty-print colon-separated file.

6. Mandatory Practical Examples¶

Example 6.1 — find + chmod (E05 Task 1)¶

Purpose¶

Find .cpp source files (case-insensitive on S) under task-3 and add write permission.

Input¶

The task-3 directory shipped with the lecture (dir-1, dir-2, dir-3 containing source*.cpp, Source*.cpp, etc.).

Code¶

find . -exec ls -l {} \;                                 # 1: inspect
find -regex './.*/\(S\|s\)ource.+\.cpp$' -exec ls -l {} \; # 2: select
find -regex './.*/\(S\|s\)ource.+\.cpp$' -exec chmod +w {} \;
find -regex './.*/\(S\|s\)ource.+\.cpp$' -exec ls -l {} \; # 3: verify

(The lecture solution uses BRE $...\|...$ because find -regex defaults to Emacs regex; alternatively find -regextype posix-extended -regex ./.*/(S|s)ource.+\.cpp$.)

Expected Output¶

-rw-rw-r-- 1 user grp 0 May  8 10:00 ./task-3/dir-1/source.txt   ← initial
-rw-rw-r-- 1 user grp 0 May  8 10:00 ./task-3/dir-1/source22.txt
...
-rwxrwxr-- 1 user grp 0 May  8 10:00 ./task-3/dir-1/document.cpp ← +w applied

Step-by-Step Explanation¶

find -regex matches the whole path, hence ./.*/.
$S\|s$ accepts both Source and source.
-exec ... \; runs the action per file (one shell call each).
chmod +w adds write to the file owner and group (depending on umask).

Real-Life HPC/CFD Meaning¶

Bulk-set permissions on hundreds of files after extracting an archive on the cluster. Writable for owner/group, read-only for others.

Written Exam Relevance¶

A typical exam phrasing: "Find all .cpp files and make them executable — give the full command." Use find -exec chmod.

Example 6.2 — grep on CSV (E05 Task 2, lecture solution)¶

# Engineering only
grep -E '^[0-9]+,([^,]+,){2}Eng.+$' employee-database.csv

# Salary > 60000  (digit-pattern — column 6 starts with [6-9])
grep '^[0-9]*,[^,]*,[^,]*,[^,]*,[^,]*,[6-9][0-9]*,.*$' employee-database.csv

# Phone starts with 345-
grep '^[0-9]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,345-.*$' employee-database.csv

# Email starts with j (column 5)
grep '^[0-9]*,[^,]*,[^,]*,[^,]*,j.*$' employee-database.csv

# Engineering OR Sales
grep -E '^[0-9]*,[^,]*,[^,]*,(Engineering|Sales),.*$' employee-database.csv

(For arithmetic comparisons such as "exactly between 60 000 and 75 000", grep's pattern is awkward — use awk.)

Real-Life Meaning¶

Quick "who matches" queries against tabular dumps without firing up Excel.

Example 6.3 — Password validation (E05 Task 3)¶

grep -E '^.{8,}$' password.txt \
 | grep -E '[A-Z]' \
 | grep -E '[a-z]' \
 | grep -E '[0-9]' \
 | grep -E '.*([!@#$%^&*].*){2}'

POSIX-class equivalent: replace [A-Z] with [[:upper:]], etc.

Example 6.4 — awk salary filter (E05 Task 4)¶

# Marketing
awk -F "," '/^[0-9]+,[^,]+,[^,]+,Marketing.*$/ {print $1" "$2" "$3}' employee-database.csv

# Salary 48000..59000
awk -F ',' '{if ($6 >= 48000 && $6 <= 59000) print $1,$2,$3}' employee-database.csv

# Marketing + tab-separated output
awk 'BEGIN {FS=",";OFS="\t"} /^[0-9]+,[^,]+,[^,]+,Marketing.*$/ {print $1,$2,$3}' employee-database.csv

Example 6.5 — sed extraction (E05 Task 5 with `script5.sed`)¶

script5.sed:

s/^([0-9]+),([^,]+),([^,]+),(Sales),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+)$/\2-\5/p

Run:

sed -E -n -f script5.sed employee_database.csv

The -n plus p flag prints only matching, transformed lines: <name>-<email>.

Example 6.6 — column (E05 Task 6)¶

column -t -s ":" /etc/passwd > new_file.txt

(The lecture writes >> (append). For a fresh file use >.)

Example 6.7 — V05 lecture awk script¶

script01.awk:

BEGIN {print "Employees with salaries greater than $69000"}

{
    if ($4 > 69000)
        print $1" "$2" "$3" $"$4" "$5;
}

END {print "Done with the printing"}

Run: awk -f script01.awk employee_database.txt. Demonstrates BEGIN/END and field arithmetic.

Example 6.8 — V05 lecture sed script¶

script02.sed:

s/^[0-9]+ +[^ ]+ [A-Za-z_]+/& Dept/
s/Engineering/Engg./g
s/^([0-9]+) ([^_]+)_([^ ]+)/\1 \3_\2/

Run: sed -E -f script02.sed employee_database.txt. Shows three substitutions: tag, replace, swap with backreferences.

Example 6.9 — diff¶

diff -u diff1.txt diff2.txt

Output (unified):

--- diff1.txt
+++ diff2.txt
@@ -1,3 +1,3 @@
 line1
-old
+new
 line3

Example 6.10 — column on a CSV¶

column -t -s , file.csv | head

7. Real HPC/CFD Workflow¶

# 1. Find every .out larger than 1G
find $WORK -type f -name "*.out" -size +1G

# 2. Extract residuals over time
grep -E "^Time =" run.log \
  | awk '{print $3, $7}' \
  | column -t > residuals.dat

# 3. Bulk parameter sweep edit
sed -i -E 's/^Re *= *.*/Re = 5000/' case_*/in.dat

# 4. Compare today's residuals against baseline
diff -u baseline_residuals.dat residuals.dat | head

# 5. Find files updated in last hour and tar them
find . -mmin -60 -type f -print0 | xargs -0 tar czf hourly.tgz

8. Exercises and Solutions¶

All E05 tasks are solved above (6.1–6.6).

Marking schemes¶

Task 1 (5 marks): 1 find regex syntax, 1 -exec, 1 chmod +w, 1 verify, 1 quoting/escaping.

Task 2 (10 marks): 2 each for the five sub-queries.

Task 3 (5 marks): 1 length, 1 upper, 1 lower, 1 digit, 1 special-char count.

Task 4 (8 marks): 2 each for Marketing filter, salary range, FS/OFS in BEGIN, combined script.

Task 5 (5 marks): 2 sed regex grouping, 2 backreferences, 1 -n + p flag.

Task 6 (4 marks): 2 for column -t -s ":", 1 input file, 1 redirection.

Common mistakes¶

Forgetting -E in grep when using +, ?, ().
Using , as separator in awk without setting FS.
sed -i 's/.../.../' (no flag) → forgets the g global.
find . -name *.cpp (unquoted) — shell expansion may break.
awk '$6 > 60000' works on whitespace-separated; on CSV it needs -F,.

Harder versions¶

Same problems but with TSV (tab-separated): awk -F'\t'.
Output to specific file and stdout: awk … | tee out.dat.

9. Written Exam Focus¶

9.1 Short Answers¶

Q. Difference between grep, sed, awk.
A. grep selects matching lines; sed transforms each line via substitutions; awk is a small language treating each line as fields, suitable for column-arithmetic.

Q. What does awk -F, '{print $3}' file.csv do?
A. Prints the third comma-separated field of every line.

Q. What is sed -i.bak?
A. Edit the file in place and keep a .bak backup of the original.

Q. How does find -exec cmd {} \; differ from ... +?
A. \; runs cmd once per match; + batches matches into one call (faster).

Q. Output of column -t -s ":" /etc/passwd | head -1?
A. A nicely aligned table: root x 0 0 root /root /bin/bash with whitespace columns.

9.2 Medium Answers¶

Q. (8 marks) Compare grep, awk, sed with one example each.

A. grep selects: grep "ERROR" run.log. awk extracts/computes: awk '$3>1000{print $1,$3}' data.txt. sed rewrites: sed -i 's/Re=1000/Re=5000/g' in.dat. grep returns matching lines unchanged; sed transforms them; awk is the most powerful — it can also work as both. In an HPC pipeline, you typically select with grep, extract with awk, fix with sed.

Q. (5 marks) Write an awk program that reads a CSV id,name,dept,salary and prints the average salary per department.

A.

BEGIN {FS=","}
NR>1 { sum[$3]+=$4; count[$3]++ }
END  { for (d in sum) printf "%s %.2f\n", d, sum[d]/count[d] }

9.3 Long Answer (12 marks)¶

Q. Discuss how find, grep, sed, awk and column cooperate in a CFD post-processing pipeline.

A.

Introduction. CFD post-processing is largely textual: residuals, timings, error reports — all in plain logs. Each Linux tool has a sharp role and they compose via pipes.

Main concept. The Unix philosophy: small tools, plain text streams, glued by pipes.

Step-by-step.

find $WORK/case_* -name run.log -print0 → list every log.
xargs -0 grep -hE "^Time = " → keep only "Time" lines.
awk '{print $3, $7}' → extract time and residual columns.
sed -E 's/[eE]\+?/×10^/g' → cosmetic fix for plotting.
column -t > residuals.dat → align into table.

Diagram. Pipeline find → grep → awk → sed → column → file. Each stage's stdin/stdout connects to the next.

Example output.

0       1.0e-3
0.001   8.7e-4
...

Real HPC/CFD link. Without these tools, you'd open the log in a GUI, copy-paste, manually delete lines — minutes per case ⇒ hours per study. Streams turn it into seconds.

Conclusion. The text-tool quintet is the workhorse of HPC scripting; mastery directly accelerates research.

9.4 Output Prediction¶

Input numbers.txt:

Q. grep -E '^15[0-9]+$' numbers.txt

A. matches 150 1500 15000 150150 150150150 (lines that are 15 followed by one or more digits — pure numeric).

Q. awk 'length($0)>=6 {print}' numbers.txt

A. lines with ≥6 chars: 150012 (no — that's 7 chars actually 1500012), 15000abc (8), 150150 (6), 150150150 (9), 2150150 (7).

9.5 Comparison¶

grep vs awk vs sed — table in 9.2.

find vs locate

	`find`	`locate`
Database	live walk	prebuilt (`updatedb`)
Speed	slower	very fast
Up-to-date	yes	depends on DB
Predicates	rich	name only

9.6 Templates¶

find template: "find <path> [-type f|d] [-name '...'] [-size +/-N(k|M|G)] [-mtime +/-N] -exec cmd {} +"

awk template: "awk -F<sep> 'BEGIN{...} pattern{action} END{...}' file"

sed template: "sed -E -i.bak 's/<re>/<repl>/g' file"

9.7 Marking Scheme — "Find .cpp and chmod +x" (5 marks)¶

1 mark: find with path.
1 mark: -name "*.cpp" (quoted).
1 mark: -type f.
1 mark: -exec chmod +x {} \; or +.
1 mark: verify with re-list.

10. Very Hard Questions¶

Beginner¶

Find all .txt files in ~. → find ~ -type f -name "*.txt".
Count lines with "ERROR" in log. → grep -c ERROR log.
Replace first foo with bar per line. → sed 's/foo/bar/' f.
Print 2nd field of CSV. → awk -F, '{print $2}' f.
Diff two files. → diff a b.

Intermediate¶

Files larger than 100 MB. → find . -size +100M.
Replace all foo with bar in place. → sed -i 's/foo/bar/g' f.
Sum column 3. → awk '{s+=$3} END{print s}' f.
Print only first match per file. → grep -m1 PAT *.log.
List unique departments. → awk -F, 'NR>1{print $3}' f | sort -u.

Hard¶

CSV with quoted commas — how to parse? → use awk -v FPAT='([^,]+)|("[^"]+")' or csvkit.
Compute median of a column. → awk '{a[NR]=$1} END{n=asort(a);print (n%2)?a[(n+1)/2]:(a[n/2]+a[n/2+1])/2}' (gawk).
Replace IPs with [redacted]. → sed -E 's/[0-9]{1,3}(\.[0-9]{1,3}){3}/[redacted]/g'.
Show 5 longest lines. → awk '{print length, $0}' f | sort -nr | head -5.
Diff two huge files only summary. → diff -q a b.

Very Hard¶

awk pivot table by department & gender. → awk -F, 'NR>1{c[$3,$5]++} END{for(k in c) print k,c[k]}'.
sed conditional: replace only if line begins with Re. → sed -E '/^Re/s/=[0-9]+/=5000/'.
find avoiding node_modules. → find . -name node_modules -prune -o -type f -print.

Deep Integration¶

Combine: search every .cpp in tree for cout not followed by endl. → grep -rEn 'cout[^;]*;[^l]*' --include='*.cpp' ..
Build a "biggest 10 directories" report. → du -sk * | sort -nr | head -10.

Coding/Command¶

awk: line numbers > 1, salary > 50000, print first 3 columns. → awk -F, 'NR>1 && $4>50000 {print $1,$2,$3}' f.
sed swap "Last_First" → "First_Last": sed -E 's/([A-Za-z]+)_([A-Za-z]+)/\2_\1/'.

Debugging¶

awk '{print $3}' file.csv prints empty for all lines. Why? → FS is whitespace by default; need -F,.
sed 's/(foo|bar)/X/g' keeps (foo|bar) literal. Why? → Need -E for ERE.

Long Written¶

(250 words) Discuss sed vs awk: when should each be used? Use Section 4.4 + 4.3.

11. Debugging and Mistake Analysis¶

Mistake	Why wrong	Correct	Explanation
`find . -name *.cpp`	shell expanded `*.cpp` first	`find . -name "*.cpp"`	quote
`grep "(a\\|b)" f`	BRE: parens literal	`grep -E "(a\\|b)" f`	use ERE
`sed -i 's/a/b/' f`	replaces only first per line	`sed -i 's/a/b/g' f`	add `g`
`awk '{$3>1000}' f`	missing `print`	`awk '$3>1000{print}'`	predicate vs action
`sed 's/+/-/' f`	`+` literal in BRE — fine, but use `-E` for ERE	`sed -E 's/\+/-/' f`	flavour
`find -exec rm -rf {} \;` typo	catastrophic	always `-print` first then `-exec`	safety
Forgot `-print0`/`-0` for filenames with spaces	breaks pipelines	use `-print0 \\| xargs -0 …`	safety
`awk` line ending `\n` confusion	recourse on `\r\n` files	`tr -d '\r'` first	DOS line endings
`column -t -s ","` not aligning	needs `column -t -s ',' -o '\|'` etc.	check version	tooling
`diff a b` huge	use `-q` brief or `-u` unified	choose wisely	scale

12. Mini Project for Mastery¶

Goal: Generate a department-by-salary report from employee_database.csv.

# 1. Extract clean rows (skip header)
awk -F, 'NR>1' employee_database.csv > clean.csv

# 2. Average salary per department
awk -F, '{sum[$4]+=$6; cnt[$4]++} END{for(d in sum) printf "%-15s %.2f\n", d, sum[d]/cnt[d]}' clean.csv \
  | sort -k2 -nr \
  | column -t > avg_salary.txt

# 3. Top earner per department
awk -F, '{
  if ($6>top[$4]) {top[$4]=$6; name[$4]=$2"_"$3}
} END {for (d in top) printf "%-15s %s %d\n", d, name[d], top[d]
}' clean.csv | column -t > top_earners.txt

# 4. Replace currency in any new export
sed -E 's/^([0-9]+,[^,]+,[^,]+,[^,]+,[^,]+,)([0-9]+)/\1\$\2/' clean.csv > with_currency.csv

Connection to exam: combines awk arithmetic, awk pattern, sed substitution, column.

13. Final Chapter Cheat Sheet¶

Item	Memorise
`find . -type f -name "*.cpp"`	files by name
`find . -size +100M`	size predicate
`find . -mtime -1`	modified today
`find -exec cmd {} +`	batched action
`grep -nE "(a\\|b)" f`	line# + ERE
`grep -rli "pat" .`	recursive, files only, case-i
`sed -i.bak 's/a/b/g'`	in-place all
`sed -n '/p/p'`	only print matches
`sed -E 's/(.+)_(.+)/\2_\1/'`	swap
`awk -F, '{print $1,$3}'`	CSV cols
`awk '{s+=$1} END{print s}'`	sum
`awk 'BEGIN{FS=",";OFS="\t"}'`	translate sep
`awk 'NR>1 && $4=="X" && $6>100'`	combined filter
`diff -u a b`	unified diff
`column -t -s ":" file`	pretty table
Trap	grep without `-E` and using `+` `()`
Top phrase	"grep selects, sed rewrites, awk computes — composed by pipes."

14. Mock Exam — Four Levels¶

Level 1 — Basic (definitions & syntax)¶

Q1. Which tool would you use to (a) locate files by size, (b) print lines containing a word, (c) replace text in-place, (d) sum a column?

Solution: (a) find -size, (b) grep, (c) sed -i, (d) awk.

Q2. What do NR and NF mean in awk?

Solution: NR = current record (line) number across input; NF = number of fields in the current line.

Q3. Write the find command that deletes all .tmp files under the current tree.

Solution: find . -type f -name "*.tmp" -delete

Q4. What is the difference between sed 's/a/b/' and sed 's/a/b/g'?

Solution: Without g only the FIRST occurrence per line is replaced; with g all occurrences on each line.

Q5. What does diff -u old.cpp new.cpp produce?

Solution: A unified diff: hunks marked @@ -l,c +l,c @@ with - lines (removed) and + lines (added) — the format used by git diff and patch.

Level 2 — Intuitive (predict the output / explain why)¶

Q1. File f contains three lines: a, ab, b. Predict: grep -c b f

Solution: 2 — lines ab and b contain a b; -c counts matching LINES, not matches.

Q2. Predict the output: echo "x:y:z" | awk -F: '{print NF, $NF}'

Solution: 3 z — three fields; $NF is the last field (NF=3 so $3).

Q3. Why does sed -n 'p' file print each line once but sed 'p' file twice?

Solution: Default cycle auto-prints the pattern space; command p prints it again → 2×. -n disables auto-print → only the explicit p remains.

Q4. find . -name "*.log" -o -name "*.out" -delete deletes only .out files. Why?

Solution: -and binds tighter than -o: the expression parses as -name "*.log" OR (-name "*.out" AND -delete). Fix with grouping: find . $ -name "*.log" -o -name "*.out" $ -delete.

Q5. awk '$1=="x"' f prints some lines of f — there is no {action}. Why does it work?

Solution: The default action is {print $0}; a pattern alone prints matching lines.

Level 3 — Hard (exam level)¶

Q1. (8 marks) From residuals.dat with columns iter rho u v p, print the iteration with the LARGEST pressure residual (column 5) and its value, using one awk command.

Solution:

awk 'NR==1 || $5>max {max=$5; it=$1} END{print it, max}' residuals.dat

Initialize on the first line, update whenever column 5 exceeds the stored max, report in END. বাংলা ইঙ্গিত: max-খোঁজার ছাঁচ মুখস্থ রাখো: NR==1 || $5>max — প্রথম লাইনে সবসময় সেট, পরে শুধু বড় হলে আপডেট।

Q2. (8 marks) Replace the value of dt (e.g. dt = 1e-4) by 5e-5 in every controlDict under cases/, keeping a .orig backup. One command.

Solution:

find cases/ -name controlDict -exec sed -i.orig -E 's/^(dt *= *).*/\15e-5/' {} +

find selects the files; sed group keeps the prefix; -i.orig writes backups; {} + batches. বাংলা ইঙ্গিত: find+sed-এর যুগলবন্দি — find দেয় কোথায়, sed দেয় কী বদলাবে; backup চাইলেই -i.bak ধাঁচ।

Q3. (8 marks) Count how many UNIQUE users appear in column 3 of a whitespace table jobs.txt (with a header line). Build the pipeline.

Solution:

awk 'NR>1 {print $3}' jobs.txt | sort -u | wc -l

NR>1 skips the header; sort -u dedups; wc -l counts. বাংলা ইঙ্গিত: "unique কতগুলো" শুনলেই sort -u | wc -l (বা sort | uniq | wc -l) — আর header টপকাতে NR>1।

Q4. (10 marks) A CSV employee.csv has name,dept,salary. Produce a per-department TOTAL salary table, formatted in aligned columns, sorted by total descending.

Solution:

awk -F, 'NR>1 {s[$2]+=$3} END{for (d in s) print d, s[d]}' employee.csv | sort -k2 -nr | column -t

An associative array accumulates per-dept totals; sort -k2 -nr orders by the numeric 2nd column; column -t aligns. বাংলা ইঙ্গিত: "per-group যোগফল" মানেই awk-এর associative array s[$2]+=$3 — এটা এই কোর্সের সবচেয়ে শক্তিশালী one-liner ছাঁচ।

Q5. (10 marks) Explain the difference between -exec cmd {} \; and -exec cmd {} +, including the process-count consequence for 10 000 files.

Solution: \; runs cmd once PER file → 10 000 processes; + appends as many paths as fit into one command line (like xargs) → a handful of processes. Same result, dramatically different cost; prefer + unless the command needs exactly one file per invocation. বাংলা ইঙ্গিত: \; = প্রতি ফাইলে নতুন process, + = একসাথে অনেক ফাইল — HPC-তে 10k ফাইলে এটা মিনিট-বনাম-সেকেন্ডের পার্থক্য।

Level 4 — Beyond the lecture (transfer + coding)¶

Q1. Your MPI job writes one log per rank: log.000 … log.255. Write ONE pipeline that prints the rank numbers whose final residual (last line, column 2) exceeds 1e-3.

Solution:

for f in log.[0-9][0-9][0-9]; do
    awk -v r="${f#log.}" 'END{if ($2+0 > 1e-3) print r}' "$f"
done

tail-free: awk's END sees the last line's fields. $2+0 forces numeric compare; ${f#log.} strips the prefix to get the rank. বাংলা ইঙ্গিত: END ব্লকে $2 মানে শেষ লাইনের দ্বিতীয় field — tail+cut না লাগিয়ে এক awk-এই কাজ; আর string-কে সংখ্যা বানাতে +0 যোগ করা awk-এর classic কৌশল।

Q2. Write an awk one-liner that converts a CFD residual log iter residual into gnuplot-ready data: skip non-numeric lines, print iter log10(residual).

Solution:

awk '$1+0==$1 && $2>0 {print $1, log($2)/log(10)}' run.log > plot.dat

$1+0==$1 is true only for numeric fields; awk has natural log only, so divide by log(10). বাংলা ইঙ্গিত: awk-এ log10 নেই — log(x)/log(10); আর "সংখ্যা কি না" পরীক্ষা $1+0==$1 — দুটোই out-of-syllabus মার্কা প্রশ্নের প্রিয় উপাদান।

Q3. Using diff in a script: write a bash snippet that runs the solver, compares out.dat with reference.dat, and exits 1 with the message "REGRESSION" if they differ by more than whitespace.

Solution:

./solver > out.dat
if ! diff -q -w out.dat reference.dat > /dev/null; then
    echo "REGRESSION" >&2
    exit 1
fi

-w ignores whitespace; -q reports only whether they differ; the exit status drives the if. বাংলা ইঙ্গিত: diff-এর exit status-ই আসল output এখানে — 0 মানে same, 1 মানে ভিন্ন; regression test-এর পুরো যুক্তি ওই status-এর উপর।

Q4. Performance thinking: grep -r "kappa" /scratch/proj is slow on a 2 TB tree. Give two ways to cut the search space using find/grep options, and one reason grep -r may still beat find+exec.

Solution: (1) Restrict file types: grep -r --include="*.cpp" --include="*.h" kappa /scratch/proj; (2) prune with find first: find /scratch/proj -name "*.cpp" -mtime -7 -exec grep -l kappa {} + (only recent sources). grep -r can still win because it's a single process doing its own traversal — no fork/exec overhead per batch and no argv-length juggling. বাংলা ইঙ্গিত: বড় ট্রিতে আগে ফাইল কমাও (include-filter, mtime), পরে content খোঁজো — I/O-ই এখানে আসল খরচ, regex নয়।

End of Chapter 7.

Chapter 7: Linux Text Tools — find, grep, awk, sed, diff, column¶

1. Chapter Overview¶

2. Basics from Zero¶

3. Hard English Made Easy¶

4. Deep Theory Explanation¶

4.1 find¶

4.2 grep¶

4.3 sed¶

4.4 awk¶

4.5 diff¶

4.6 column¶

4.7 Diagram explanations from V05¶

5. Command / Syntax / Code Breakdown¶

find . -name "*.txt" -type f¶

find ... -exec chmod +w {} \;¶

grep -E "(foo|bar)" file¶

grep -nA 2 -B 1 ERROR run.log¶

sed -E -i.bak 's/^Re *= *[0-9]+/Re = 5000/' file¶

awk -F, 'NR>1 && $4=="Engineering" && $6>60000 {print $1,$2,$3}' employee.csv¶

diff -u a b¶

column -t -s ":" /etc/passwd > new_file.txt¶

6. Mandatory Practical Examples¶

Example 6.1 — find + chmod (E05 Task 1)¶

Purpose¶

Input¶

Code¶

Expected Output¶

Step-by-Step Explanation¶

Real-Life HPC/CFD Meaning¶

Written Exam Relevance¶

Example 6.2 — grep on CSV (E05 Task 2, lecture solution)¶

Real-Life Meaning¶

Example 6.3 — Password validation (E05 Task 3)¶

Example 6.4 — awk salary filter (E05 Task 4)¶

Example 6.5 — sed extraction (E05 Task 5 with script5.sed)¶

Example 6.6 — column (E05 Task 6)¶

Example 6.7 — V05 lecture awk script¶

Example 6.8 — V05 lecture sed script¶

Example 6.9 — diff¶

Example 6.10 — column on a CSV¶

7. Real HPC/CFD Workflow¶

8. Exercises and Solutions¶

Marking schemes¶

Common mistakes¶

Harder versions¶

9. Written Exam Focus¶

9.1 Short Answers¶

9.2 Medium Answers¶

9.3 Long Answer (12 marks)¶

9.4 Output Prediction¶

9.5 Comparison¶

9.6 Templates¶

9.7 Marking Scheme — "Find .cpp and chmod +x" (5 marks)¶

10. Very Hard Questions¶

Beginner¶

Intermediate¶

Hard¶

Very Hard¶

Deep Integration¶

Coding/Command¶

Debugging¶

Long Written¶

11. Debugging and Mistake Analysis¶

12. Mini Project for Mastery¶

13. Final Chapter Cheat Sheet¶

14. Mock Exam — Four Levels¶

Level 1 — Basic (definitions & syntax)¶

Level 2 — Intuitive (predict the output / explain why)¶

Level 3 — Hard (exam level)¶

Level 4 — Beyond the lecture (transfer + coding)¶

4.1 `find`¶

4.2 `grep`¶

4.3 `sed`¶

4.4 `awk`¶

4.5 `diff`¶

4.6 `column`¶

`find . -name "*.txt" -type f`¶

`find ... -exec chmod +w {} \;`¶

`grep -E "(foo|bar)" file`¶

`grep -nA 2 -B 1 ERROR run.log`¶

`sed -E -i.bak 's/^Re = [0-9]+/Re = 5000/' file`¶

`awk -F, 'NR>1 && $4=="Engineering" && $6>60000 {print $1,$2,$3}' employee.csv`¶

`diff -u a b`¶

`column -t -s ":" /etc/passwd > new_file.txt`¶

Example 6.5 — sed extraction (E05 Task 5 with `script5.sed`)¶