Text processing

Before you read. A web server has been misbehaving. Its log has 1.2 million lines in it. Somewhere in there is the answer.

You cannot read 1.2 million lines. You cannot open it in an editor. And you do not yet know what you are looking for, so you cannot search for it either.

How do you get from a million lines to a handful without knowing in advance which handful you want?

By asking questions of the shape of the data rather than its content. Which addresses appear most often. Which status codes. How many of each. Those answers fit on a screen, and one of them will tell you what to search for.

That is one pipeline, four commands long, and it is the single most useful thing in this lesson. The rest is the vocabulary that lets you build variations of it.

The example log throughout is small enough to check by eye, so you can verify that each command did what the prose claims. Everything works identically at a million lines.

Some words you will need

regular expression

A pattern language for describing text. grep and sed both take one.

field

One column of structured text, decided by a separator. Field 1 of a log line is usually the address.

delimiter

The character separating fields. A space in a log, a colon in /etc/passwd, a comma in a CSV.

filter

A command that reads stdin, changes it, and writes stdout. Everything here is one, which is why they compose.

What breaks without this

Investigation stops at the size of the file. Anything you cannot read by eye becomes unusable, which is every log on every real system.

You answer the wrong question. “Is the error in there?” is a yes-or-no with little value. “Which client is causing them and how many?” is the useful one, and it needs counting rather than searching.

Reports become manual work. Anything you do by reading and tallying is something a four-command pipeline does correctly every time.

grep: finding lines

# Debian 13 (trixie), x86_64
$ cd /tmp; grep 403 access.log
10.0.0.14 - - [07/Aug/2026:09:14:11] "GET /admin HTTP/1.1" 403 199
10.0.0.14 - - [07/Aug/2026:09:16:02] "GET /admin HTTP/1.1" 403 199
10.0.0.14 - - [07/Aug/2026:09:17:01] "GET /admin HTTP/1.1" 403 199
10.0.0.14 - - [07/Aug/2026:09:19:55] "GET /admin HTTP/1.1" 403 199

grep prints lines that match. That is the whole idea, and the flags are where the value is:

-v is the one people forget and it is half the job. Excluding the noise is frequently faster than describing the signal: grep -v ' 200 ' access.log gives you every request that was not a success.

-C 3 is what you want when reading a stack trace or a service log, because the line that matched is rarely the whole story.

Enough regular expression to be useful

Anchors are the highest-value two characters here. ^root finds the root account in /etc/passwd; plain root also finds every account whose home is under /root and every comment mentioning it.

grep -E turns on extended expressions, which add +, ?, |, and () without backslashes: grep -E 'error|warning|fatal' logfile is the everyday use.

The pipeline that answers most questions

# Debian 13 (trixie), x86_64
$ cd /tmp; echo "--- who is hitting us, most first ---"; cut -d" " -f1 access.log | sort | uniq -c | sort -rn
--- who is hitting us, most first ---
      5 10.0.0.14
      2 10.0.0.9
      2 10.0.0.31
      1 10.0.0.7

Reading it as a sentence: take one column, group the same values together, count each group, show the biggest first.

grep    '^[0-9]\{3\}-[0-9]\{4\}$'    # BRE: braces escaped to be special
grep -E '^[0-9]{3}-[0-9]{4}$'        # ERE: braces are special already
grep -P '^\d{3}-\d{4}$'              # PCRE: \d, lookarounds, non-greedy

cut, and its limitation

cut pulls out columns. -d sets the delimiter, -f picks the fields:

cut -d: -f1,7 /etc/passwd     # login name and shell
cut -d, -f2 data.csv          # second column of a CSV
cut -c1-8 file                # characters 1 to 8, ignoring fields entirely

cut treats every delimiter as significant, which is fine for /etc/passwd where the colons are exact and a real problem for text aligned with spaces. Two spaces between columns means an empty field between them, and -f2 returns nothing.

There is no cut flag that fixes this. awk is the answer, and that is largely why awk exists.

awk: fields, done properly

The second command below filters and prints in one pass: $8 == 403 selects the lines, {print $1} prints the address from each.

# Debian 13 (trixie), x86_64
$ cd /tmp; echo '--- awk picks fields by number ---'; awk '{print $1, $8}' access.log | head -4; echo '--- and can filter as it goes ---'; awk '$8 == 403 {print $1}' access.log | sort -u
--- awk picks fields by number ---
10.0.0.14 200
10.0.0.9 200
10.0.0.14 403
10.0.0.31 200
--- and can filter as it goes ---
10.0.0.14

$1 is the first field, $8 the eighth, $0 the whole line, NF the number of fields. Any run of whitespace separates them, so the cut problem above does not arise.

An awk program is pattern { action }, and either half can be left out:

-F sets the separator for text that is not whitespace-delimited: awk -F: '{print $1}' /etc/passwd.

And awk can count as it goes, which is where it stops being a better cut:

# Debian 13 (trixie), x86_64
$ cd /tmp; echo '--- sed replaces ---'; sed 's/10\.0\.0\./10.0.0.x/' access.log | head -3; echo '--- how many bytes served in total ---'; awk '{total += $9} END {print total}' access.log
--- sed replaces ---
10.0.0.x14 - - [07/Aug/2026:09:14:02] "GET /index.html HTTP/1.1" 200 5120
10.0.0.x9 - - [07/Aug/2026:09:14:07] "GET /style.css HTTP/1.1" 200 812
10.0.0.x14 - - [07/Aug/2026:09:14:11] "GET /admin HTTP/1.1" 403 199
--- how many bytes served in total ---
17194

END { } runs once after the last line, which is how you accumulate a total. BEGIN { } runs once before the first, for headers and setting FS.

awk '{a[$1]++} END {for (k in a) print a[k], k}' replaces the whole sort | uniq -c pipeline with one pass and no sorting. The associative array is awk’s real party trick, and it is the version that scales to a file too large to sort.

sed: rewriting

sed 's/old/new/' substitutes. The pieces:

In the capture above, 10\.0\.0\. has escaped dots, because an unescaped . matches any character. The dots are the point of the exercise: 10.0.0. would also match 100000x and a great deal else.

sed -i deserves suspicion. It does not edit in place: it writes a new file and renames it over the original, which breaks hard links, changes the inode, and needs free space. Run it without -i first and read the output. sed -i.bak keeps a copy, which costs nothing and has saved a great many afternoons.

Everything else, briefly

tr -d '\r' earns its place. A config file edited on Windows has \r\n line endings, and the trailing carriage return becomes part of the last value on every line, so PORT=8080\r is not 8080, and the error message will not mention it. file thefile says with CRLF line terminators and dos2unix fixes it properly.

xargs bridges the gap between commands that produce lines and commands that take arguments:

grep -rl 'old.example.com' /etc | xargs sed -i.bak 's/old/new/g'

-r on xargs stops it running at all when the input is empty, which prevents a command running with no arguments and doing something unintended. -0 with find -print0 handles filenames containing spaces.

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

awk '{print $8}' access.log | sort | uniq -c | sort -rn

du -h --max-depth=1 /var | sort -h | tail -20

lsof +L1

journalctl -u sshd --since '24 hours ago' \
  | grep 'Failed password' \
  | awk '{print $(NF-3)}' \
  | sort | uniq -c | sort -rn

comm -23 <(sort a.txt) <(sort b.txt)

Across distributions

The tools are GNU coreutils, GNU grep, GNU sed, and GNU awk on every distribution this exam covers, so they behave identically. Two things to know anyway:

The sed -i difference is the one that catches people writing on a Mac and deploying to Linux, and it fails loudly rather than subtly, which is the better kind of incompatibility.

Prove it

Build pipelines one stage at a time, checking after each:

# Stage 1: is the field right
cut -d" " -f1 access.log | head

# Stage 2: add the sort, confirm grouping
cut -d" " -f1 access.log | sort | head

# Stage 3: add the count
cut -d" " -f1 access.log | sort | uniq -c | head

# Stage 4: order it
cut -d" " -f1 access.log | sort | uniq -c | sort -rn | head

| head after each stage keeps the output readable while you check that the stage did what you meant. Building the whole thing and running it once produces output you cannot verify, and a wrong field number looks exactly like a correct one until you check.

What trips people up

1. uniq without sort first

uniq compares adjacent lines only. Unsorted input gives a count of 1 for almost everything, and the output looks plausible.

Always sort | uniq -c. Or sort -u when you want deduplication without counts.

2. A dot matches anything

grep 10.0.0.1 also matches 100.0.0.1, 10x0y0z1, and more. In a regular expression . is any character.

grep -F 10.0.0.1 for a literal string, or escape them: 10\.0\.0\.1.

.* is greedy in the same way and matches as much as it can, which is why s/".*"/X/ on a line with two quoted strings replaces everything between the first and last quote rather than each pair.

3. cut cannot cope with runs of spaces

Two spaces mean an empty field between them, so -f2 on aligned output returns nothing.

awk '{print $2}' treats any run of whitespace as one separator. Use cut for exact delimiters like /etc/passwd, awk for anything spaced by eye.

4. sed -i on a file you have not backed up

It rewrites the file with no undo, and a regex that matches more than you intended does so on every line at once.

Run it without -i and read the output. Then sed -i.bak.

5. Building the whole pipeline before testing any of it

A six-stage pipeline that produces nothing gives no clue which stage is at fault. Build up, checking with head after each.

Work it through

A web server is slow. The log is 1.2 million lines. Nobody knows what changed.

Reason through the order before reading on.

Do not search yet. You do not know what you are looking for, and grep requires knowing. Start with distributions, which need no hypothesis.

What status codes are we returning?

awk '{print $8}' access.log | sort | uniq -c | sort -rn

A normal shape is mostly 200s with a scattering of 304s and 404s. A large count of 500s means the application; a large count of 403s means something is being refused repeatedly; a large count of 499s or 408s means clients giving up, which points at slowness rather than errors.

Who is generating the traffic?

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

If one address has a large share, that is a scraper, a broken client, or a health check that somebody set to one second. If it is flat, the load is genuine and this is a capacity question rather than an incident.

What are they asking for?

awk '{print $7}' access.log | sort | uniq -c | sort -rn | head -20

One expensive endpoint dominating is the common answer, and it names the thing to fix.

Now you know enough to search. Cross the two:

awk '$1 == "10.0.0.14" {print $7, $8}' access.log | sort | uniq -c | sort -rn | head

which tells you what that specific client was doing and whether it was being served or refused.

And check the time distribution, because “slow since when” is the question everyone asks next:

awk '{print substr($4, 14, 5)}' access.log | sort | uniq -c

substr pulls the hour and minute out of the timestamp field. A count per minute turns “slow today” into “started at 09:15”, and 09:15 is something you can correlate with a deployment.

Now the point worth extracting: every one of those is the same pipeline with a different field number. Extract a column, group, count, order. Learn it once and the only thing you vary is which column and which filter, which is why this lesson is short and covers more ground than it looks.

And the habit: start with distributions, not searches. A search needs a hypothesis; a distribution produces one. Beginning with grep means you find only what you already suspected, which on a problem nobody understands is nothing.

Try it

Optional, on any machine. /var/log/ has real material.

1. grep -c bash /etc/passwd, then grep -v nologin /etc/passwd | wc -l.
2. cut -d: -f1,7 /etc/passwd | head. Then cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn, how many accounts use each shell.
3. awk -F: '$3 >= 1000 {print $1}' /etc/passwd, every real human account.
4. awk -F: '{total += $3} END {print total}' /etc/passwd, which is meaningless and proves the mechanism.
5. du -h --max-depth=1 /var 2>/dev/null | sort -h | tail.
6. Build a pipeline one stage at a time with | head after each, and notice how much easier it is to spot a wrong field number.
7. echo "hello world" | tr 'a-z' 'A-Z', then tr -s ' ' on text with runs of spaces.

Verification step. You have it when, given an unfamiliar structured file, you can produce a count of the most common value in any column without looking anything up.

Check yourself

awk '{print $8}' access.log | sort | uniq -c | sort -rn

References

• grep(1) - Linux man-pages project. Accessed 2026-08-07.
• sed(1) - Linux man-pages project. Accessed 2026-08-07.
• cut(1) - Linux man-pages project. Accessed 2026-08-07.
• sort(1) - Linux man-pages project. Accessed 2026-08-07.
• uniq(1) - Linux man-pages project. Accessed 2026-08-07.
• regex(7) - Linux man-pages project. Accessed 2026-08-07.
• GNU Awk User’s Guide - GNU Project. Accessed 2026-08-07.

Every block above with a distribution and architecture header was captured by running the command on a Debian 13 (trixie) container. Blocks without one are illustrative.