Chapter 6: Regular Expressions and Globbing¶
Source slides:
L04_Regexp.pdf. Exercise:E04_Regexp.pdf(+ solutions). Test files inL04/L04/E04_files/(task-1.txt,task-2.txt,task-4.txt,dir-1/,dir-2/,dir-3/).
1. Chapter Overview¶
Two pattern languages dominate Linux:
- Globbing (filename wildcards used by the shell):
*,?,[abc],{a,b},[!abc]— expanded by the shell before the command runs. - Regular expressions (regex): a richer language used inside programs (
grep,sed,awk, Vim, JavaScript, Python). Two flavours of POSIX regex: BRE (Basic) and ERE (Extended), plus PCRE (Perl-Compatible).
Why it matters in HPC/CFD: every CFD log file is gigabytes of text. The pattern languages let you find the line where the residual diverged, replace a parameter in 200 cases, list only header files with a digit suffix.
What the examiner asks (very high frequency):
- "Difference between regex and globbing."
- "Difference between BRE and ERE."
- "Write a regex that matches a German phone number / email / IP address."
- "Predict matches of a given regex against given text."
- "Write a glob that lists
.hfiles indir-1anddir-3ending in a digit."
What you must master for top grade:
- The 12 metacharacters:
. * + ? ^ $ [] {} () | \. - POSIX character classes:
[:alpha:] [:alnum:] [:digit:] [:space:] [:upper:] [:lower:] [:punct:]. - Anchors:
^,$,\<,\>,\b. - Quantifiers:
*,+,?,{n},{n,},{n,m}(greedy / lazy). - Groups & alternation.
- Glob basics:
*,?,[…],{a,b}brace expansion. - The four solutions of E04 (phone, email, glob, /etc/passwd).
2. Basics from Zero¶
A glob is a small wildcard the shell expands. Type ls *.txt — the shell turns *.txt into the full list of matching filenames before running ls.
A regex is a string that describes a pattern. [a-z]+@[a-z]+\.[a-z]{2,} describes a simple email. Programs like grep test each input line against the pattern and act (print, replace, count).
The difference matters: globs operate on filenames, regex operates on text. They look similar (* in both) but mean different things — * in glob means "any string of any chars", in regex means "zero or more of the previous". A regex *.txt is invalid — you'd want .*\.txt.
বাংলায়: Glob আর regex দেখতে একরকম হলেও সম্পূর্ণ আলাদা দুটো ভাষা। Glob shell নিজে expand করে ফাইলের নামের উপর, command চালু হওয়ার আগেই; আর regex চলে program-এর ভেতরে, ফাইলের লেখার (content) উপর।
*glob-এ মানে "যেকোনো string", কিন্তু regex-এ মানে "ঠিক আগের জিনিসটা শূন্য বা তার বেশি বার" — এই এক পার্থক্যই পরীক্ষার এক নম্বর প্রশ্ন।
Real-life analogy. A glob is the search box of Windows Explorer. A regex is the Find & Replace … dialog with regex enabled in your IDE.
Real-life HPC use.
- Glob:
ls case_*/{run.log,residuals.dat}. - Regex:
grep -E "diverg(ed|ing)" run.log,sed -i -E 's/Re *= *[0-9]+/Re = 5000/' input.dat.
বাংলায়: বাস্তবে দুটোই একসাথে লাগে: glob দিয়ে বেছে নাও কোন ফাইলগুলো (যেমন সব case directory-র log), আর regex দিয়ে খোঁজো সেই ফাইলগুলোর ভেতরে কী লেখা আছে (যেমন কোথায় residual diverge করল)। HPC-তে এই জুটিটাই দৈনন্দিন কাজের ভিত্তি।
What if you misunderstand? You write grep ?.txt thinking glob — but ? here is a regex meaning "previous optional", and grep reads stdin not files. You write ls .*\.txt — the shell does not understand \., so ls lists nothing as expected. Wrong tool, wrong syntax.
বাংলায়: ভুল tool বাছলে অনেক সময় কোনো error-ও আসে না — শুধু চুপচাপ ভুল ফলাফল আসে। যেমন regex হিসেবে
*.txtলেখাটাই invalid (regex-এ*-এর আগে একটা atom লাগে), ঠিকটা.*\.txt। তাই আগে নিজেকে প্রশ্ন করো: আমি কি ফাইলের নাম match করছি (glob, shell-এর কাজ) নাকি লেখার ভেতরে খুঁজছি (regex, program-এর কাজ)?
3. Hard English Made Easy¶
| Hard Term | Simple English | বাংলা | Example |
|---|---|---|---|
| Regular expression | A pattern that matches text | টেক্সটের প্যাটার্ন | [a-z]+ |
| Globbing | Filename wildcard expansion in the shell | শেলে ফাইল-নাম প্যাটার্ন | *.txt |
| Wildcard | A symbol standing for many chars | অনেক অক্ষরকে প্রতিনিধিত্বকারী চিহ্ন | * ? |
| Metacharacter | A regex symbol with special meaning | বিশেষ অর্থযুক্ত চিহ্ন | . * + ? |
| Anchor | Position marker | অবস্থান চিহ্ন | ^, $ |
| Quantifier | Says how many to match | সংখ্যা নির্দেশক | * + ? {n} |
| Greedy | Match as much as possible | সর্বাধিক ম্যাচ | .* |
| Lazy / non-greedy | Match as little as possible | সর্বনিম্ন ম্যাচ | .*? |
| Character class | Set of allowed chars | অক্ষর শ্রেণী | [abc] |
| Negated class | All except listed chars | বাদে অক্ষর | [^abc] |
| Alternation | OR | অথবা | cat\|dog |
| Capture group | Numbered () group | নম্বরযুক্ত গ্রুপ | (abc) |
| Backreference | Reuse captured text | পূর্বের গ্রুপ পুনঃব্যবহার | \1, $1 |
| BRE / ERE / PCRE | POSIX Basic / Extended / Perl-style | রেগেক্স ভ্যারিয়ান্ট | grep / grep -E / grep -P |
Glob brace {a,b} |
Expands to a, b separately |
একাধিক বিকল্পে এক্সপ্যান্ড | f.{cpp,h} |
| Locale-dependent class | [A-Z] may include accents |
লোকাল-নির্ভর শ্রেণী | LANG-sensitive |
4. Deep Theory Explanation¶
4.1 Globbing (shell)¶
The shell sees the unquoted *, ?, […], {…,…} and expands them to a list of matching filenames before the command runs. If nothing matches, in default Bash the unexpanded pattern is passed literally — you can change with shopt -s nullglob or failglob.
| Pattern | Meaning |
|---|---|
* |
any string (incl. empty), excluding / |
? |
exactly one char |
[abc] |
one of a/b/c |
[a-z] |
range |
[!abc] or [^abc] |
not a/b/c |
{a,b,c} |
brace expansion (comma-separated) |
** |
recursive (with globstar enabled in bash) |
~ |
home dir (special, not a glob) |
.* |
hidden files (only if you write the dot literally) |
Examples:
{a,b,c} is brace expansion — done by the shell before globbing.
* does not match leading . by default, hence ls * will not show .bashrc.
Who expands what — the most examinable picture of this chapter:
COMMAND LINE: grep -E 'diverg(ed|ing)' case_*/run.log
└──────┬───────┘ └──────┬─────┘
regex (quoted, glob (unquoted,
shell ignores it) shell expands it)
│
┌─────────────────────────────┐ │
│ STEP 1 — the SHELL │ ◄───────────────────┘
│ expands the GLOB against │ case_*/run.log
│ FILENAMES, before exec: │ ──► case_1/run.log case_2/run.log
└──────────────┬──────────────┘
▼ starts the program with the expanded argument list
┌─────────────────────────────┐
│ STEP 2 — the PROGRAM (grep) │ applies the REGEX to every LINE
│ receives: │ of the files' CONTENT:
│ diverg(ed|ing) │ "...solution diverged at step 512..."
│ case_1/run.log │ ────────► match
│ case_2/run.log │
└─────────────────────────────┘
The quotes around the regex stop the shell from touching it.
বাংলায়: মনে রেখো, glob expand করে shell — command টা নয়।
ls *.txtচালালে ls কখনোই*.txtদেখে না, সে দেখে আগে থেকেই expand হওয়া নামের তালিকা। কিছু match না করলে default bash pattern-টা literal আকারেই পাঠিয়ে দেয় (nullglob দিয়ে বদলানো যায়)। আর{a,b}brace expansion glob-ও নয় — সেটা আরও আগে ঘটে, ফাইল থাকুক বা না থাকুক।
4.2 Regex variants¶
| Variant | Triggered by | Meta-chars |
|---|---|---|
| BRE | grep, sed (default) |
., *, ^, $, [], \(\), \{n,m\}, \| (some) |
| ERE | grep -E, sed -E, egrep, awk |
., *, +, ?, ^, $, (), {}, | |
| PCRE | grep -P, perl, python |
adds \d \w \s, lookarounds, *? lazy, etc. |
| Vim | inside Vim | \(, \|, \<, \>; "magic" levels |
In BRE, (, ), {, }, |, ?, + are literal unless escaped (\(, \|). In ERE, those are special by default and escape gives literal.
বাংলায়: BRE আর ERE-র পার্থক্য একটাই নিয়মে ধরা যায়: BRE-তে ( ) { } + ? এবং alternation-এর bar সাধারণ অক্ষর, special বানাতে backslash লাগে; ERE-তে উল্টো — এগুলো জন্ম থেকেই special, literal বানাতে backslash লাগে।
grepমানে BRE,grep -Eবা awk মানে ERE। পরীক্ষায় "কেনgrep '(a|b)'কিছু পাচ্ছে না" — উত্তর এটাই।
4.3 Core syntax¶
| Item | BRE | ERE | PCRE | Meaning |
|---|---|---|---|---|
| Any char | . |
. |
. |
any except newline |
| 0+ repeats | * |
* |
* |
0 or more of previous |
| 1+ | \+ |
+ |
+ |
1 or more |
| 0 or 1 | \? |
? |
? |
optional |
Repeat {n} |
\{n\} |
{n} |
{n} |
exact |
{n,m} |
\{n,m\} |
{n,m} |
{n,m} |
range |
| Alternation | \| (GNU) |
| |
| |
OR |
| Group | \(...\) |
(...) |
(...) |
capture |
| Backref | \1 |
\1 |
\1 |
re-use |
| Anchor start | ^ |
^ |
^ |
start of line |
| Anchor end | $ |
$ |
$ |
end of line |
| Word boundary | \<, \> |
same | \b |
word edges |
| Char class | [abc] |
same | same | one of |
| Negated class | [^abc] |
same | same | not |
| POSIX class | [[:alpha:]] |
same | same | letters |
| Shorthand | n/a | n/a | \d \w \s |
digit/word/space |
| Lazy | n/a | n/a | *? |
non-greedy |
Common POSIX classes inside []:
[:alpha:] letters · [:alnum:] letters+digits · [:digit:] digits · [:space:] whitespace · [:upper:], [:lower:] · [:xdigit:] hex · [:punct:] punctuation · [:cntrl:] control.
Use as [[:alpha:][:digit:]_] (note double brackets).
Anatomy of a bracket expression:
[ ^ a - z 0 - 9 [:space:] ]
│ │ └─┬─┘ └─┬─┘ └───┬───┘ │
│ │ │ │ │ └── closing bracket of the expression
│ │ │ │ └──────── POSIX class: space, tab, newline, …
│ │ │ └───────────────── range: every char from '0' to '9'
│ │ └──────────────────────── range: every char from 'a' to 'z'
│ └───────────────────────────── '^' in FIRST position = negate the set
└─────────────────────────────── a bracket expression matches EXACTLY
ONE character of the input
Inside [...] most metacharacters lose their power: [.*+] is literal . * +
A literal ']' must come first ( []abc] ); a literal '-' first or last.
বাংলায়: Quantifier সবসময় ঠিক আগের atom-টার উপর কাজ করে, পুরো pattern-এর উপর নয়:
ab+মানে a তারপর এক বা একাধিক b, কিন্তু(ab)+মানে ab ব্লকটা একাধিক বার। আর POSIX class কখনো একা লেখা যায় না — bracket-এর ভেতরে bracket লাগবেই:[[:digit:]], শুধু[:digit:]লিখলে ভুল। Bracket-এর ভেতরে.বা*এর কোনো special ক্ষমতা নেই — এটাও প্রিয় exam-trap।
4.4 Greediness vs laziness¶
a.*b against axxbyyb matches axxbyyb (greedy). a.*?b matches axxb (lazy). Lazy needs PCRE.
বাংলায়: Greedy quantifier আগে যতটা সম্ভব গিলে নেয়, তারপর বাকি pattern মেলাতে দরকার হলে পিছিয়ে আসে (backtracking)। Lazy (
*?) ঠিক উল্টো — যত কম সম্ভব নিয়ে শুরু করে, দরকার হলে বাড়ায়। এক লাইনে দুটো quote-করা string থাকলে".*"প্রথম quote থেকে শেষ quote পর্যন্ত সব ধরে ফেলে, আর".*?"শুধু প্রথমটা ধরে। Lazy টা PCRE-only — BRE/ERE-তে নেই, সেখানে negated class দিয়ে কাজ চালাতে হয় (4.9 দেখো)।
4.5 Anchors¶
^start of line,$end. Use(?m)flag for multi-line in PCRE.\<start of word,\>end of word (BRE/ERE).\bword boundary,\Bnon-boundary (PCRE/Vim).
বাংলায়: Anchor কোনো অক্ষর match করে না — শুধু অবস্থান (position) যাচাই করে, তাই এদের width শূন্য।
^লাইনের শুরু,$লাইনের শেষ,\<\>শব্দের সীমানা। Anchor বাদ দিলে regex লাইনের যেকোনো জায়গায় match করবে — email বা phone validation-এর প্রশ্নে anchor ভুলে যাওয়াই সবচেয়ে বড় ফাঁদ, কারণ তখন আবর্জনাসহ লাইনও pass করে যায়।
4.6 Backreferences¶
\1, \2, … refer to the n-th group. Useful in substitutions:
4.7 Globbing vs regex — the comparison table¶
| Feature | Globbing | Regex |
|---|---|---|
| Used by | Shell | Programs (grep/sed/awk/Vim) |
| Operates on | Filenames | Text/lines |
* means |
any string (no /) |
0+ of previous |
? means |
one char | optional previous |
[abc] |
char class | char class |
{a,b} |
brace expansion | not in BRE/ERE; works in PCRE alternation (a|b) |
| Anchors | none | ^, $ |
| Power | small | very large |
বাংলায়: তুলনার টেবিলটা মুখস্থ রাখো: glob কাজ করে ফাইলের নামের উপর, regex কাজ করে লাইনের ভেতরের লেখার উপর; glob-এর
?মানে ঠিক একটা অক্ষর, আর regex-এর?মানে আগের জিনিসটা optional। পরীক্ষায় এক লাইনের সংজ্ঞা চাইলে লেখো: "Globs name files; regex matches text."
4.8 Diagrams¶
The lecture shows three diagrams worth recreating:
- Glob
dir-{1,3}/*[0-9].h→ expands todir-1/header9.h dir-3/05header1.h …. - Regex
\+49 [0-9]{3}-[0-9]{3}-[0-9]{4}annotated piece-by-piece. - /etc/passwd line annotated with seven
:-separated fields →^[^:]+:[^:]+:…$regex template.
4.9 Formal semantics — quantifiers, classes, anchors, and the engine¶
Quantifiers as set definitions¶
Let \(L(r)\) be the set of strings the atom \(r\) can match, and let \(L(r)^k\) denote the concatenation of \(k\) copies, with \(L(r)^0 = \{\varepsilon\}\) (the empty string). The interval quantifier is a finite union:
and the three shorthands are special cases of it:
Worked numeric example. How many distinct strings does the anchored ERE ^[0-9]{2,4}$ accept? Each position chooses independently from 10 digits, and the lengths 2, 3, 4 are disjoint cases:
Same logic for letters: ^[a-z]{3}$ accepts \(26^3 = 17576\) strings.
Character classes as sets¶
A bracket expression denotes a set of characters; a range is defined through the character codes:
A negated class such as [^abc] is the complement with respect to the full alphabet \(\Sigma\):
POSIX classes are named sets: \([[:digit:]] = \{0, \dots, 9\}\) with \(|\cdot| = 10\); \([[:alnum:]] = [[:alpha:]] \cup [[:digit:]]\). This is why class size questions are pure counting: [0-9a-f] has \(10 + 6 = 16\) members (hexadecimal digits).
Anchors as zero-width assertions¶
A match inside a line \(s = c_1 c_2 \dots c_n\) is a pair of positions \((i, j)\) with \(0 \le i \le j \le n\); the matched substring is \(c_{i+1} \dots c_j\). Anchors consume no characters — they only constrain the positions: ^ forces \(i = 0\), $ forces \(j = n\). Consequently ^$ matches exactly the empty line (\(i = j = 0 = n\)), and an unanchored regex may match anywhere inside the line.
Greedy vs lazy — fully worked example¶
Input line with its character indices:
The quotes sit at indices 7, 13, 21, 24.
- Greedy
".*": the first"matches at index 7;.*consumes everything up to index 24, then backtracks just enough for the final"— which is the last quote (24). One match, spanning indices 7–24:"alpha", id = "42"(18 characters). - Lazy
".*?"(PCRE only):.*?starts with zero characters and grows only on demand, so it stops at the nearest closing quote (13). First match:"alpha"(indices 7–13). Scanning resumes at 14, giving a second match"42"(indices 21–24). - The portable trick: the ERE
"[^"]*"produces the same two matches as lazy — the negated class can never step over a quote. This is the standard way to "fake" laziness in grep/sed/awk.
BRE vs ERE — the precise backslash table¶
| Construct | BRE (grep, sed) |
ERE (grep -E, egrep, awk, sed -E) |
|---|---|---|
| Group | \( \) |
( ) |
| Interval | \{m,n\} |
{m,n} |
| One or more | \+ (GNU extension) |
+ |
| Optional | \? (GNU extension) |
? |
| Alternation | backslash-bar (GNU extension) | bare bar |
Literal ( ) { } + ? |
written plain | must be escaped: \( \+ … |
. * ^ $ [ ] |
special unescaped | special unescaped |
Backreference \1 … \9 |
yes | yes (GNU) |
side by side: BRE ERE
alternation: a\|b a|b
group + repeat: \(ab\)\{2,3\} (ab){2,3}
one or more: a\+ (GNU) a+
literal plus sign: + \+
Memory rule: in BRE the backslash switches a character ON (makes it special); in ERE the backslash switches it OFF (makes it literal). The set . * ^ $ [ ] is special in both flavours.
How the engine matches — a step-by-step trace¶
A backtracking engine (grep, PCRE) tries every start position from left to right; at each position it walks through the pattern, and greedy quantifiers give characters back on failure. Trace of [a-z]+[0-9]\. against mesh42.dat:
Pattern: [a-z]+ [0-9] \. Input: m e s h 4 2 . d a t
index: 0 1 2 3 4 5 6 7 8 9
Attempt at start position 0:
┌──────┬─────────┬───────────────┬──────────────────────────────────────────┐
│ Step │ Pattern │ Input │ Action │
├──────┼─────────┼───────────────┼──────────────────────────────────────────┤
│ 1 │ [a-z]+ │ 0..3 "mesh" │ greedy: eats m,e,s,h; '4' not in [a-z] │
│ 2 │ [0-9] │ 4 '4' │ match — advance to index 5 │
│ 3 │ \. │ 5 '2' │ FAIL: '2' is not '.' → backtrack │
│ 4 │ [a-z]+ │ 0..2 "mes" │ gives one char back ('h') │
│ 5 │ [0-9] │ 3 'h' │ FAIL → give back 's', then 'e', then… │
│ 6 │ │ │ [a-z]+ exhausted (needs at least 1 char) │
│ │ │ │ → attempt at position 0 FAILS │
└──────┴─────────┴───────────────┴──────────────────────────────────────────┘
The engine then retries from positions 1, 2, …, 9 — every attempt fails
(no letter run is followed by EXACTLY one digit and then a dot).
Result: NO match in "mesh42.dat"
Contrast: in "mesh4.dat" → [a-z]+ = "mesh", [0-9] = '4', \. = '.' →
MATCH "mesh4." at indices 0..5, found on the very first attempt.
বাংলায়: Regex engine বাঁ থেকে ডানে প্রতিটা start position চেষ্টা করে, আর greedy quantifier আগে বেশি খেয়ে পরে ফেরত দেয় — এটাই backtracking। উপরের trace-এ দেখো
[0-9]মানে ঠিক একটা digit, তাই mesh42-এর দুটো digit কখনোই pattern-এ আঁটে না — quantifier-এর সংখ্যা-নিয়ন্ত্রণ কতটা কড়া, এই উদাহরণটা সেটাই শেখায়। পরীক্ষায় "কেন match হলো না" ধরনের প্রশ্নে এভাবে ধাপে ধাপে trace লিখলে পুরো নম্বর।
5. Command / Syntax / Code Breakdown¶
Glob examples¶
ls *.txt # only .txt in this dir
ls **/*.cpp # recursive (needs shopt -s globstar)
ls dir-?/*.h # one-char dir name
ls {dir-1,dir-2}/*.{cpp,h,txt} # brace + glob
ls dir-{1,3}/*[0-9].h # ending with a digit
ls dir-3/*[A-Z]*.h # has at least one uppercase
ls .[!.]* # hidden files except '.' and '..'
shopt -s nullglob # no match → empty list
echo {1..5} # 1 2 3 4 5
echo {a..e} # a b c d e
echo {01..03} # 01 02 03
echo file_{2024,2025}.{cpp,h} # full Cartesian product
Regex with grep¶
grep "Re = 1000" log # BRE plain
grep -E "diverg(ed|ing)" log # ERE
grep -P "\d{3}-\d{4}" file # PCRE
grep -i pattern file # case-insensitive
grep -v pattern file # invert
grep -n pattern file # line numbers
grep -c pattern file # count
grep -r pattern dir # recursive
grep -l pattern *.log # filenames only
grep -oE "[A-Z]+" file # only matched part
grep -A 3 -B 1 pattern file # context after / before
Regex with sed (in-place edit)¶
Regex inside Vim¶
In Vim, \+ \? \( \) need backslash by default ("magic" mode); use \v (very magic) for ERE-like syntax: :%s/\v(Re|Pr) = \d+/&/g.
বাংলায়: একই regex তিন জায়গায় তিন রকম পোশাক পরে: grep-এ BRE (escape লাগে), grep -E-তে ERE (escape লাগে না), আর Vim-এ আবার BRE-ঘেঁষা নিজস্ব নিয়ম। পরীক্ষায় কোন tool-এ কোন flavour — এটা গুলিয়ে ফেলাই সবচেয়ে কমন ভুল।
6. Mandatory Practical Examples¶
Example 6.1 — Phone numbers (E04 Task 1)¶
Purpose. Validate phone numbers that look like +49 030-123-4567 (country code + 3-3-4 digits separated by -).
Input (task-1.txt)
+49 030-123-4567
+1 555-555-5555
+44 020-7946-0958 <- wrong grouping
03012345678 <- no country code
+91 080 -123-4567
Code / Command
ERE regex (use grep -E or regex101):
Or with shorthand (PCRE):
For a specific country code (+49):
Expected Output. Matches: +49 030-123-4567, +1 555-555-5555. Doesn't match the rest.
Step-by-Step Explanation
\+literal plus.[0-9]+one or more digits (country code).- literal space.
[0-9]{3}-[0-9]{3}-[0-9]{4}the 3-3-4 grouping.
বাংলায়: Validation regex-এর মূলমন্ত্র: প্রতিটা অংশকে আলাদা টুকরায় ভাঙো — literal
+(escape করা), country code, space, তারপর 3-3-4 digit গ্রুপ।{3}মানে ঠিক তিনটা — কম-বেশি হলেই match fail, তাই ভুল grouping-এর লাইনগুলো বাদ পড়ে।
Real-Life HPC/CFD Meaning. Form-validation, log scrubbing, anonymising data exports.
Written Exam Relevance. Almost identical phrasing appears in the exercise; expect a variation (licence plate, IP address) asking for the regex.
Example 6.2 — Email validation (E04 Task 2)¶
POSIX:
PCRE:
(The lecture note misses the \ in .[[:alpha:]]{2,}$; it should be \.[[:alpha:]]{2,}$. Mention this in your written answer to score the trap mark.)
Step-by-Step Explanation
^start of line,$end.[[:alnum:]_.]+prefix: letters, digits,_,..@literal at-sign.[[:alpha:]]+domain.\.literal dot.[[:alpha:]]{2,}TLD with 2 or more letters.
Example 6.3 — Globbing (E04 Task 3)¶
ls *.txt # 1
ls {dir-1,dir-2}/*{.txt,.cpp,.h} # 2
ls {dir-1,dir-3}/*[0-9].h # 3
ls dir-3/*[A-Z]*.h # 4
The lecture's exact answer.
Example 6.4 — /etc/passwd regex (E04 Task 4)¶
/etc/passwd line: user:password:UID:GID:GECOS:home:shell.
- GID ≥ 100:
^[^:]+:[^:]+:[^:]+:[1-9][0-9]{2,}:[^:]*:[^:]+:.*$ - Home under /home:
^[^:]+:[^:]+:[^:]+:[^:]+:[^:]*:\/home.+:.*$ - Shell
/bin/bash:^[^:]+:[^:]+:[^:]+:[^:]+:[^:]*:[^:]+:\/bin\/bash
(Caveat: question (1) says "≥ 100" and the regex [1-9][0-9]{2,} matches numbers with 3 or more digits starting with a nonzero digit — i.e. ≥ 100, including ≥ 1000. Mention this nuance for top marks.)
Apply with grep
grep -E "^[^:]+:[^:]+:[^:]+:[1-9][0-9]{2,}:[^:]*:[^:]+:.*$" /etc/passwd
grep -E "^([^:]*:){5}/home" /etc/passwd
grep -E "/bin/bash$" /etc/passwd
বাংলায়:
[^:]+মানে "colon ছাড়া যেকোনো কিছু" — field-ভিত্তিক ফাইলে এক একটা field টপকানোর সেরা কৌশল। আরও ছোট রূপ:^([^:]*:){5}মানে "প্রথম পাঁচটা field পেরিয়ে যাও"। এই দুটো idiom জানা থাকলে passwd-জাতীয় যেকোনো প্রশ্ন এক মিনিটে নামে।
Example 6.5 — Glob recursion¶
7. Real HPC/CFD Workflow¶
# 1. Find all CFD case dirs
ls -d case_*/ # glob
# 2. Find inputs that already use Re=5000
grep -lE "^Re *= *5000\b" case_*/in.dat
# 3. Bulk-rename Re=1000 → Re=5000 in 200 cases
sed -i -E 's/^(Re *= *)1000/\15000/' case_*/in.dat
# 4. Find every line where the residual diverged
grep -nE "diverg(ed|ing)|nan|inf" case_*/run.log
# 5. List header files that end in a digit (lecture exercise pattern)
ls inc/*[0-9].h
বাংলায়: লক্ষ করো workflow-টা glob আর regex-এর যুগলবন্দি: glob (case_/) ঠিক করে কোন ফাইলগুলোতে, regex ঠিক করে ফাইলের ভেতরের কোন লেখায়* কাজ হবে। sed-এর
\1backreference-টা পরীক্ষার hot topic — group ধরে রেখে শুধু সংখ্যাটা বদলানো।
8. Exercises and Solutions¶
The four E04 tasks are fully solved above. For each:
- Task 1 — Phone — ERE solution
\+[0-9]+ [0-9]{3}-[0-9]{3}-[0-9]{4}. Marking: 1 mark for\+, 1 for[0-9]+, 1 for first[0-9]{3}-, 1 for second[0-9]{3}-, 1 for[0-9]{4}. - Task 2 — Email —
^[[:alnum:]_.]+@[[:alpha:]]+\.[[:alpha:]]{2,}$. Marking: 1 for anchors, 1 for prefix class, 1 for@, 1 for domain, 1 for\., 1 for{2,}. - Task 3 — Globbing — see 6.3. Marking: 4 × 2 marks for each pattern.
- Task 4 — passwd — see 6.4. Marking: 3 × 3 marks (1 each for anchor, field skipping, target match).
বাংলায়:
^মানে লাইনের শুরু,$মানে শেষ,[...]মানে অক্ষর শ্রেণী,+মানে এক বা তার বেশি।
Common mistakes.
- Missing
^/$→ matches anywhere on the line, lets bad input pass. - Missing
\.(using.) → matches any char, including "no dot at all". - Forgetting
\+is escape-required in BRE. - POSIX class single brackets:
[:alpha:]alone is wrong; you need[[:alpha:]].
Harder versions.
- Phone: also accept parentheses like
+49 (0)30-123-4567. Hint:\+[0-9]+( ?\([0-9]+\))? [0-9]{3}-[0-9]{3}-[0-9]{4}. - Email: forbid leading dot and require domain length ≥ 2:
^[A-Za-z0-9_][A-Za-z0-9_.]*@[A-Za-z0-9-]{2,}\.[A-Za-z]{2,}$.
9. Written Exam Focus¶
9.1 Short Answers¶
Q. Difference between regex and globbing.
A. Globbing is filename pattern matching done by the shell before a command runs (*, ?, […], {…}). Regex is a richer text-pattern language used by programs (grep/sed/awk/Vim) with anchors, quantifiers, classes, groups, backrefs.
Q. What does * mean in a glob vs a regex?
A. Glob: any string of any characters (excluding /). Regex: zero or more of the previous atom.
Q. What does ^abc$ match?
A. Lines containing exactly abc.
Q. Difference between BRE and ERE.
A. In BRE, (){}|+? are literal unless escaped with \. In ERE they are metacharacters by default. grep is BRE; grep -E (or egrep) is ERE.
Q. Why use [[:alpha:]] instead of [A-Za-z]?
A. It is locale-aware: [A-Z] may not include accented letters under some locales; [[:alpha:]] always covers letters.
9.2 Medium Answers¶
Q. (8 marks) Compare globbing and regex with examples. When do you use each?
A. Globbing patterns are interpreted by the shell to expand into filenames before the command runs (ls *.txt, cp *.cpp dst/). They are simple: *, ?, [abc], {a,b}. Regex is a richer language used by tools that read text streams (grep, sed, awk, Vim). It supports anchors ^$, quantifiers *+?{n,m}, alternation |, groups (), backrefs \1, character classes […] and POSIX classes [[:alpha:]]. Use globs for "give me files matching"; use regex for "find/replace text inside files". Example: ls case_*/*.log | xargs grep -E "(diverg|nan|inf)".
Q. (5 marks) Construct a regex for a German licence plate "AA 123" (1–3 letters, space, 1–4 digits).
A. ERE: ^[A-Z]{1,3} [0-9]{1,4}$. With locale-safe classes: ^[[:upper:]]{1,3} [[:digit:]]{1,4}$.
9.3 Long Answer (12 marks)¶
Q. Discuss the structure of regular expressions and how they interact with shell tools, using examples from grep, sed, and Vim.
A.
Introduction. A regex is a language for describing text patterns. POSIX defines BRE (used by grep, sed) and ERE (used by egrep, awk). PCRE adds shorthand and lookarounds.
Main concept. Patterns combine atoms (., char, class), quantifiers (* + ? {n,m}), anchors (^, $, \<, \>), groups (()), and alternation (|).
Step-by-step. Take an HPC log line: [INFO] step=120 residual=2.3e-04. To extract step and residual:
grep -oE returns matches; sed -E "s/.../\2 \3/" extracts groups; awk -F= parses fields.
Example — replace Re=1000 with Re=5000 in 200 case files:
Vim integration. :%s/\<Re\>\s*=\s*1000/Re = 5000/gc. Vim's "very magic" mode \v removes the need for many backslashes.
Real HPC/CFD link. Bulk-edit cases, log analysis, scrubbing PII before publishing data.
Conclusion. Regex is the universal pattern language; combined with globs (filename selection) and shell tools (grep/sed/awk) it forms the textual backbone of HPC scripting.
9.4 Output Prediction¶
Input task-1.txt:
Regex \+[0-9]+ [0-9]{3}-[0-9]{3}-[0-9]{4} → matches only the first line.
Glob ls dir-3/*[A-Z]*.h against HEADER05.h header9.h Project537.cpp → matches HEADER05.h only.
9.5 Comparison¶
Glob vs Regex — table in section 4.
BRE vs ERE
| BRE | ERE | |
|---|---|---|
(, ) |
literal | group |
{, } |
literal | quantifier |
+ |
literal | 1+ |
? |
literal | 0/1 |
\| |
literal (GNU adds \\\|) |
alternation |
| Escape gives | meta | literal |
9.6 Templates¶
Regex template: "^[allowed]+@[allowed]+\.[tld]{2,}$ for emails. Anchor + class + quantifier + literal + class + quantifier."
Glob template: "{dir-1,dir-3}/*[0-9].h — brace expansion + wildcard + class."
9.7 Marking Scheme — "Phone regex" (5 marks)¶
- 1 mark: anchor / no-anchor decision.
- 1 mark:
\+literal plus. - 1 mark:
[0-9]+country code. - 1 mark: literal space.
- 1 mark: 3-3-4 grouping with
-.
10. Very Hard Questions¶
Beginner
- What does
*.txtmatch in a shell? → All files ending in.txtin the current dir. - What does
^abcmatch? → Lines starting withabc. - Glob for any single character? →
?. - POSIX class for digits? →
[[:digit:]]. - Regex for "letters followed by a space"? →
[[:alpha:]]+.
Intermediate
- List
.hand.cppin dir-1 + dir-2. →ls {dir-1,dir-2}/*.{h,cpp}. - Regex for IPv4 (loose)? →
([0-9]{1,3}\.){3}[0-9]{1,3}. - Replace tabs with spaces (sed)? →
sed 's/\t/ /g'. - Whole-word match for
Re? →\<Re\>(BRE/ERE) or\bRe\b(PCRE). - Glob for hidden files? →
.[!.]*orls -A.
Hard
- Match lines that contain neither foo nor bar. →
^(?!.*(foo|bar)).*$(PCRE lookahead). - Phone with optional country code. →
(\+[0-9]+ )?[0-9]{3}-[0-9]{3}-[0-9]{4}. - Glob matching files NOT starting with
tmp. → enable extglob:ls !(tmp*). - Replace scientific-notation numbers by 0.0. →
sed -E 's/[0-9]*\.[0-9]+[eE][+-]?[0-9]+/0.0/g'. - Match a date
YYYY-MM-DD. →[0-9]{4}-[0-9]{2}-[0-9]{2}.
Very Hard
- Why is
ls *.txtfaster thanfind . -name "*.txt"? → The glob is expanded by the shell with one readdir of the current dir; find walks the whole tree recursively. - Regex for a strict IPv4 (0–255 per octet). →
((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?). - How to negate a glob? → extglob
!(pattern).
Deep Integration
- Pipeline that prints filenames whose content matches a regex:
grep -lE 'PATTERN' *.log. →-llists matching files; the glob picks candidates. - Why test on regex101.com before
sed -i? → In-place edit is destructive; verify the pattern interactively first.
Coding/Command
- Find files modified in the last day and list those containing "ERROR". →
find . -mtime -1 -type f -name '*.log' -exec grep -l ERROR {} +. - Vim: replace whole-word
oldNamewithnewNamein all open buffers. →:bufdo %s/\<oldName\>/newName/ge | update.
Debugging
grep "(foo|bar)" filereturns nothing. Why? → BRE:()and|are literal; usegrep -E "(foo|bar)" file.- Glob
ls *.[ch]lists nothing. Why? → No matching files; bash passes the pattern literally;shopt -s nullglobmakes it expand to an empty list.
Long Written
- (250 words) Discuss the use of regex and glob together in an HPC log-analysis pipeline. Use the patterns from 6.4 + 9.2.
11. Debugging and Mistake Analysis¶
| Mistake | Why wrong | Correct | Explanation |
|---|---|---|---|
grep ?.txt |
? is glob; grep wants regex |
ls ?.txt (glob) or grep '.\.txt' |
wrong tool |
name@domain.com matches nameXdomainXcom |
. matches any char |
name@domain\.com |
escape literal dot |
Forgot ^/$ in email regex |
matches partial garbage | add anchors | full-string match |
sed 's/+/-/g' surprises |
+ literal in BRE, meta in ERE |
sed -E 's/\+/-/g' |
flavour matters |
ls dir/*[A-Z]*.h returns nothing |
no caps in filenames | check filenames | locale/content issue |
\d in plain grep |
BRE doesn't support \d |
[0-9] or grep -P |
flavour |
find . -name *.txt |
shell expanded the glob first | find . -name "*.txt" |
quote it |
Greedy .* swallows too much |
greedy by default | .*? (PCRE) or [^"]* |
reduce greediness |
বাংলায়: এই টেবিলের সবচেয়ে দামি দুটো ভুল: (১)
find . -name *.txt-এ quote না দেওয়া — shell আগে glob expand করে ফেলে find ভুল argument পায়; (২) BRE-তে\dবা()ব্যবহার — grep চুপচাপ literal ধরে নেয়, error-ও দেয় না, ফল শুধু "match নেই"। পরীক্ষায় "কেন কাজ করছে না" প্রশ্নের উত্তর প্রায়ই এই দুটোর একটা।
12. Mini Project for Mastery¶
Goal: Build a tiny "log reaper" that finds and counts the most common errors across all HPC logs.
shopt -s globstar
grep -hE "(ERROR|WARN|FATAL|nan|inf)" **/*.log \
| awk -F: '{print $1}' \
| sort | uniq -c | sort -nr | head
Followed by a sed-based fix:
Connection to exam: uses globbing (**/*.log), regex ((ERROR|WARN|FATAL|nan|inf)), and sed -E substitution — three exam topics in five lines.
13. Final Chapter Cheat Sheet¶
| Item | Memorise |
|---|---|
Glob * |
any string |
Glob ? |
one char |
Glob [abc] |
char class |
Glob {a,b,c} |
brace expansion |
| Hidden glob | .[!.]* |
Regex . |
any char |
Regex * |
0+ previous |
Regex + (ERE) |
1+ previous |
Regex ? (ERE) |
0/1 |
Regex {n,m} |
n to m |
| Regex anchors | ^, $, \<, \>, \b |
| POSIX class | [[:alpha:]] |
| Backreference | \1 |
| BRE escapes | \(\)\{\}\|\+\? |
| ERE | escapes make literal |
| Vim "very magic" | \v |
| Trap | confusing glob * with regex * |
| Top phrase | "Globs name files; regex matches text — different alphabets, different power." |
14. Mock Exam — Four Levels¶
Level 1 — Basic (definitions & syntax)¶
Q1. Write a glob that lists all .dat files in the current directory.
Solution: ls *.dat
Q2. What does the regex colou?r match?
Solution: color and colour — ? makes the previous atom (u) optional (ERE/PCRE).
Q3. Write the regex anchor pair that forces a whole-line match.
Solution: ^pattern$.
Q4. Glob: list files a.txt, b.txt … any single-letter name with .txt.
Solution: ls ?.txt
Q5. What does [^0-9] match?
Solution: Any one character that is NOT a digit (negated class).
Level 2 — Intuitive (predict / explain why)¶
Q1. Directory contains mesh1.h mesh12.h meshA.h. Predict: ls mesh[0-9].h
Solution: Only mesh1.h. The class matches exactly ONE digit-character; mesh12.h has two chars between "mesh" and ".h".
Q2. Why does grep "mesh+" file find the literal text mesh+ but grep -E "mesh+" finds mesh, meshh, ...?
Solution: In BRE + is literal; in ERE it's a quantifier meaning "one or more of the previous atom" (here h).
Q3. echo case_{1..3}/run_{a,b}.log — predict the output.
Solution: case_1/run_a.log case_1/run_b.log case_2/run_a.log case_2/run_b.log case_3/run_a.log case_3/run_b.log — brace expansion is a pure text Cartesian product; no files need to exist.
Q4. A file contains aaa. How many matches does grep -o 'a*' report and why is the answer surprising?
Solution: One match aaa (greedy: takes all three). (At end-of-line a zero-length match is possible but grep -o doesn't print empty matches.) Greedy quantifiers take the longest run.
Q5. Why does ls *.log fail with "No such file or directory" in an empty dir, while ls works?
Solution: With no match, bash passes the literal string *.log to ls, which looks for a file literally named *.log → error. nullglob changes this.
Level 3 — Hard (exam level)¶
Q1. (6 marks) Write an ERE that matches a simulation time stamp of the form t=12.5s or t=0.0125s (decimal required), and reject t=12s.
Solution: t=[0-9]+\.[0-9]+s — \. forces the decimal point, both sides need at least one digit.
বাংলা ইঙ্গিত: "decimal বাধ্যতামূলক" মানেই \. literal dot-এর দুই পাশে [0-9]+ — dot escape ভুললেই সব নম্বর শেষ।
Q2. (8 marks) Using ONE grep command, count how many case directories (named case_001 … case_999) contain the word DIVERGED in their run.log.
Solution: grep -l "DIVERGED" case_[0-9][0-9][0-9]/run.log | wc -l — -l prints each matching file once; glob restricts to 3-digit case dirs; wc -l counts.
বাংলা ইঙ্গিত: "কতগুলো ফাইল" শুনলেই -l | wc -l; -c দিলে প্রতি ফাইলে match-সংখ্যা আসে — সেটা ভুল উত্তর।
Q3. (8 marks) Transform every line Re = <number> to Reynolds: <number> in all in.dat files (in-place), preserving the number. Give the sed command.
Solution: sed -i -E 's/^Re *= *([0-9]+)/Reynolds: \1/' case_*/in.dat — group captures the number, \1 reuses it.
বাংলা ইঙ্গিত: "preserving" শব্দটা দেখলেই capture group () + backreference \1 লাগবে।
Q4. (10 marks) Explain precisely, step by step, what the shell and grep each do in: grep -E "^d" dir-*/listing.txt
Solution: (1) bash sees dir-*/listing.txt, scans the cwd, and expands it to all existing paths like dir-1/listing.txt dir-2/listing.txt; (2) bash executes grep with those expanded paths as arguments; (3) grep, per file, applies the ERE ^d to each line and prints lines starting with d, prefixed by the filename (multiple files). If no dir matches, grep gets the literal pattern path and errors.
বাংলা ইঙ্গিত: মূল পয়েন্ট: glob expand করে SHELL, grep regex চালায় ফাইলের content-এ — দুটো ধাপ আলাদা করে লিখলেই পুরো নম্বর।
Q5. (10 marks) Write a single ERE for a valid 24-hour time HH:MM (00:00–23:59). Explain each alternative branch.
Solution: ^([01][0-9]|2[0-3]):[0-5][0-9]$ — branch 1: [01][0-9] covers 00–19; branch 2: 2[0-3] covers 20–23; minutes [0-5][0-9] covers 00–59.
বাংলা ইঙ্গিত: range-এর গণিত regex জানে না — তোমাকেই সংখ্যা-পরিসরকে digit-pattern-এ ভাঙতে হবে; এটাই IPv4 প্রশ্নেরও কৌশল।
Level 4 — Beyond the lecture (transfer + coding)¶
Q1. A bash script must verify its argument is a positive float in scientific notation (e.g. 1.5e-3). Write the test using [[ =~ ]].
Solution:
if [[ "$1" =~ ^[0-9]+\.?[0-9]*([eE][+-]?[0-9]+)?$ ]]; then
echo "valid"
else
echo "invalid" >&2; exit 1
fi
[[ =~ ]] uses ERE; the exponent group is optional, mantissa allows 1, 1., 1.5.
বাংলা ইঙ্গিত: bash-এর =~ ERE বোঝে কিন্তু pattern-টা quote করলে literal হয়ে যায় — unquoted রাখাই নিয়ম।
Q2. Your C++ solver writes logs where floats are sometimes 1,5 (comma — German locale). Write one command that converts decimal commas to dots ONLY in numbers (not in CSV separators) in results.csv, given fields are separated by ;.
Solution: sed -E -i 's/([0-9]),([0-9])/\1.\2/g' results.csv — the groups require a digit on both sides, so ;-separators and text commas are untouched. Run twice or use perl for overlapping matches like 1,2,3 if a digit chain shares boundaries: perl -pe 's/(?<=[0-9]),(?=[0-9])/./g' results.csv handles all cases in one pass via lookarounds.
বাংলা ইঙ্গিত: sed-এর match-গুলো overlap করে না — 1,2,3-এর মাঝের কমা দুটো এক pass-এ ধরতে lookaround (perl/PCRE) লাগে; এই subtlety-টাই প্রশ্নের আসল ফাঁদ।
Q3. Combine find + glob knowledge: delete all object files (*.o) older than 7 days anywhere under the project, but print each name before deleting. Why is a plain glob not enough?
Solution: find . -name "*.o" -type f -mtime +7 -print -delete. A glob can't filter by age and **/*.o requires globstar + still can't test mtime; find predicates evaluate left→right, so -print before -delete logs every removal.
বাংলা ইঙ্গিত: glob শুধু নাম দেখে; সময়/সাইজ/টাইপ শর্ত মানেই find — আর -delete সবসময় শেষে, নাহলে আগে মুছে পরে print করার সুযোগই থাকে না।
Q4. An exam-grade pipeline: from squeue output (columns: JOBID PARTITION NAME USER ST TIME NODES), print the JOBIDs of YOUR pending (PD) jobs, sorted numerically descending. Use grep+awk+sort.
Solution:
or robustly with awk alone:squeue | awk -v u="$USER" '$4==u && $5=="PD" {print $1}' | sort -nr. The awk version compares whole fields, immune to spacing.
বাংলা ইঙ্গিত: column-ভিত্তিক ডেটায় grep-এর চেয়ে awk-এর field comparison নিরাপদ — $4==u কখনো অন্য কলামে ভুল match করে না; regex হলে username-এর substring বিপদ ঘটাতে পারত।
End of Chapter 6.