Explaining a shell pipeline one stage at a time

A pipeline is a compressed argument. Each stage makes a claim about the data and hands the result along, but on the page all of that collapses into a single line, and the reader sees the conclusion without the reasoning. Pasted under “this finds the most common 404s”, that line gives them something to copy and nothing to adapt when their log format differs from yours.

The fix is mechanical: build the command in front of the reader, one stage per step, with the output after each. Everything below was run on Debian 12 (GNU coreutils 9.1, mawk, bash) and again on macOS 26 with bash 3.2; the outputs come from the Debian run, and the differences that turned up are noted.

State the question, then show the finished command

Start with what the pipeline answers, in ordinary words: which missing pages are requested most often? Then give the whole thing at once, so the reader knows where this is heading:

awk '$9 == 404 { print $7 }' access.log | sort | uniq -c | sort -rn | head -n 2

A reader in a hurry takes that and leaves, which is fine. Everyone else now has a map. In a Unix pipeline, each program’s standard output becomes the next program’s standard input, so the order of the stages is the order of your explanation.

Hand the reader the input

Most readers do not have your server’s logs, so give them a small file that the post itself creates. A heredoc does that in one paste:

cat > access.log <<'EOF'
203.0.113.7 - - [02/Nov/2026:09:14:01 +0000] "GET /old-pricing HTTP/1.1" 404 512
198.51.100.4 - - [02/Nov/2026:09:14:03 +0000] "GET /docs/install HTTP/1.1" 200 8120
203.0.113.9 - - [02/Nov/2026:09:14:08 +0000] "GET /favicon.png HTTP/1.1" 404 512
198.51.100.4 - - [02/Nov/2026:09:14:11 +0000] "GET /old-pricing HTTP/1.1" 404 512
192.0.2.33 - - [02/Nov/2026:09:14:15 +0000] "GET /robots.txt HTTP/1.1" 200 404
203.0.113.7 - - [02/Nov/2026:09:14:19 +0000] "GET /wp-login.php HTTP/1.1" 404 512
192.0.2.33 - - [02/Nov/2026:09:14:22 +0000] "GET /old-pricing HTTP/1.1" 404 512
198.51.100.8 - - [02/Nov/2026:09:14:27 +0000] "GET /favicon.png HTTP/1.1" 404 512
203.0.113.9 - - [02/Nov/2026:09:14:31 +0000] "GET /docs/install HTTP/1.1" 200 8120
192.0.2.40 - - [02/Nov/2026:09:14:36 +0000] "GET /old-pricing HTTP/1.1" 404 512
EOF

Ten lines can be checked by eye, which is the point. The file also contains two deliberate traps, because sample data should include the awkward cases your pipeline is written to handle. The reasons for testing all of this on a clean machine are in How to write a tutorial someone else can reproduce.

Add one stage, show one output

Keep a strict rhythm: one sentence on what the new stage does, the command with everything so far, the output. First, keep the 404s and print only the path:

awk '$9 == 404 { print $7 }' access.log

/old-pricing
/favicon.png
/old-pricing
/wp-login.php
/old-pricing
/favicon.png
/old-pricing

Say what the numbers mean, because $9 is opaque to anyone who has not counted the fields. Awk splits on whitespace, so the path lands in field 7 and the status code in field 9. Next, put identical paths side by side:

awk '$9 == 404 { print $7 }' access.log | sort

/favicon.png
/favicon.png
/old-pricing
/old-pricing
/old-pricing
/old-pricing
/wp-login.php

Collapse each run of identical lines into one, with a count:

awk '$9 == 404 { print $7 }' access.log | sort | uniq -c

      2 /favicon.png
      4 /old-pricing
      1 /wp-login.php

Order by that count, highest first, then keep the top two:

awk '$9 == 404 { print $7 }' access.log | sort | uniq -c | sort -rn | head -n 2

      4 /old-pricing
      2 /favicon.png

Repeating the growing command feels wasteful, but readers can paste any step and see exactly where their screen stopped matching yours. Keep each command and its output in separate blocks, for the reasons in Code blocks readers can copy and paste safely.

Explain why by removing a stage

Readers adapting the command need to know why each stage is there, and the quickest way to show that is to break the pipeline on purpose. Take out the first sort:

awk '$9 == 404 { print $7 }' access.log | uniq -c

      1 /old-pricing
      1 /favicon.png
      1 /old-pricing
      1 /wp-login.php
      1 /old-pricing
      1 /favicon.png
      1 /old-pricing

uniq only compares each line with the one before it, so unsorted input gives a count of one for almost everything. That is the first trap: the repeated paths in the sample are deliberately not adjacent.

The second trap answers the question experienced readers will ask: why not filter with grep?

grep -c 404 access.log
awk '$9 == 404' access.log | wc -l

8
7

The extra match is the /robots.txt line, a successful request whose response happened to be 404 bytes long. grep looks at the whole line, while awk tests the one field that holds the status.

Flag what will differ on the reader’s machine

Mention the differences you actually found, close to where they appear. In this example, uniq -c pads its counts to seven columns on Debian and four on macOS, and wc -l pads its number on macOS but not on Debian. The content is the same, but a reader comparing screens may wonder.

Ties are the other quiet problem. Our sample has none at the top, but real data will, and sort -rn reverses its fallback comparison as well, so tied lines come out in reverse alphabetical order. If the output must be stable, say so and sort on both keys:

printf '2 /b\n2 /a\n1 /c\n' | sort -k1,1nr -k2,2

2 /a
2 /b
1 /c

The text ordering itself depends on the locale, so a post that shows exact sorted output of mixed-case names should set LC_ALL=C on that sort, or warn that the order may vary.

Say what happens when a stage fails

A pipeline’s exit status is normally that of its last command, so a failure at the start disappears:

awk '$9 == 404 { print $7 }' missing.log | sort | uniq -c
echo "exit: $?"

Awk complains about the missing file on standard error, in wording that varies by implementation, and then the shell prints exit: 0. In a script, that reads as success with no results. Run set -o pipefail first, in bash, and the same two lines print exit: 2.

If you recommend pipefail, mention its catch, because the head in our pipeline is the sort of stage that triggers it. When head has what it needs, it exits, and a stage still writing to it may be killed by SIGPIPE. You can see it directly:

yes | head -n 1
echo "${PIPESTATUS[@]}"

y
141 0

That 141 is 128 plus 13, the signal number for SIGPIPE, on yes, while head reports 0. With pipefail set, the whole pipeline would report 141 even though nothing went wrong. PIPESTATUS is a bash feature; zsh spells it pipestatus.

End by giving the reader a way to inspect their own run. Dropping tee stage1.txt between any two stages saves what passed through without disturbing the rest, which is how they will debug it when their log format is not yours. If you write several of these walkthroughs, a shared tag collects them on a topic page; the docs list the paths.