Shell redirection and pipes

Before you read. A command has just printed four thousand lines. You needed six of them.

You could scroll. You could run it again into a file and open the file. Or you could hand its output straight to something that finds the six lines, without the four thousand ever reaching the screen.

Here is the question underneath, and it is the one that makes Linux feel different once it lands: why does a system full of small, single-purpose commands beat one large program that does everything?

Because the small ones can be joined together, and the joining costs nothing. A program that counts lines does not need to know how to search, and a program that searches does not need to know how to count, because the output of one becomes the input of the other.

That is the whole of the Unix philosophy, and it is two characters of syntax.

Some words you will need

standard input (stdin)

Where a program reads from when nobody says otherwise. Your keyboard, normally.

standard output (stdout)

Where its normal results go. The terminal, normally.

standard error (stderr)

Where its complaints go. Also the terminal, and that is why they look the same when they are not.

redirection

Pointing one of those three at a file instead.

pipe

Connecting one program's stdout to the next one's stdin.

exit status

A number every command returns when it finishes. Zero means success.

What breaks without this

You cannot capture what happened. Every log, every report, every “send me the output” ends with redirection, and getting it wrong means the errors are missing from the file you sent.

Scripts lie about failing. A pipeline reports the status of its last command only, so a script can check for failure and find none while the important step went wrong.

Everything takes more steps than it needs. Without pipes, every intermediate result becomes a temporary file you create, use, and forget to delete.

Three channels

Errors go somewhere different from results. That is the fact this whole lesson rests on, and nothing on screen tells you, because both end up in the same terminal.

Separate them and it becomes obvious:

# Debian 13 (trixie), x86_64
$ ls /etc/hostname /nosuchfile > out.txt 2> err.txt; echo "--- out.txt (stdout) ---"; cat out.txt; echo "--- err.txt (stderr) ---"; cat err.txt
--- out.txt (stdout) ---
/etc/hostname
--- err.txt (stderr) ---
ls: cannot access '/nosuchfile': No such file or directory

One command, two destinations. The filename that worked went to stdout; the complaint about the one that did not went to stderr.

The design reason is worth having. If errors went to stdout, then ls *.txt > list.txt on a directory with a problem would put the error message into list.txt, and whatever read that file next would treat it as a filename. Keeping them apart means a pipeline carries data and a human sees the problems, which is exactly what you want.

Replace or append

# Debian 13 (trixie), x86_64
$ echo first > log.txt; echo second > log.txt; echo "--- after two > redirects ---"; cat log.txt; echo third >> log.txt; echo "--- after >> ---"; cat log.txt
--- after two > redirects ---
second
--- after >> ---
second
third

> truncates before it writes. first was gone before second was written, and nothing warned. The file is emptied the moment the shell sets up the redirection, which is also why cmd > file on a file that cmd is reading produces an empty file rather than an error.

>> appends. For anything that accumulates (a log, a report a script adds to) that is what you want, and reaching for > out of habit is how a day’s output becomes one line.

Both at once, and the order that matters

# Debian 13 (trixie), x86_64
$ echo "--- redirect both, the right way ---"; ls /etc/hostname /nosuchfile > both.txt 2>&1; cat both.txt; echo "--- and the wrong order ---"; ls /etc/hostname /nosuchfile 2>&1 > wrong.txt; echo "(the error above went to the terminal, not the file)"; cat wrong.txt
--- redirect both, the right way ---
ls: cannot access '/nosuchfile': No such file or directory
/etc/hostname
--- and the wrong order ---
ls: cannot access '/nosuchfile': No such file or directory
(the error above went to the terminal, not the file)
/etc/hostname

sort file -o file              # sort knows about this and handles it
sed -i 's/x/y/' file           # in-place, via a temporary file
sponge < file > file           # from moreutils; soaks up stdin first
grep x file > tmp && mv tmp file

set -o noclobber
echo hi > existing.txt      # bash: existing.txt: cannot overwrite existing file
echo hi >| existing.txt     # the explicit override

Pipes

A pipe connects one command’s stdout to the next one’s stdin. No temporary file, and the two run at the same time rather than one after the other.

# Debian 13 (trixie), x86_64
$ echo "--- how many accounts ---"; cat /etc/passwd | wc -l; echo "--- same thing, one process fewer ---"; wc -l < /etc/passwd; echo "--- tee writes and passes through ---"; grep bash /etc/passwd | tee shells.txt | wc -l; echo "--- and the file has it too ---"; wc -l shells.txt
--- how many accounts ---
18
--- same thing, one process fewer ---
18
--- tee writes and passes through ---
1
--- and the file has it too ---
1 shells.txt

Three things there.

cat file | wc -l works and is a tell. wc -l < file and wc -l file do the same job without an extra process. It is called a useless use of cat, nobody will die, and it is worth not doing.

tee splits the stream, writing to a file and passing everything through. It is what you want when a pipeline should also leave a record, and | sudo tee /etc/somefile is the standard answer to the redirection-and-sudo problem from lesson 06, because then the privileged process is the one doing the writing.

Pipes run concurrently. command | head -5 does not wait for the first command to finish; head exits after five lines and the first command is stopped with a broken pipe. That is why head on a huge file is instant.

Exit status

Every command returns a number. Zero means success, anything else means a specific failure, and $? holds the last one.

# Debian 13 (trixie), x86_64
$ grep -q root /etc/passwd; echo "found root, exit status $?"; grep -q notarealuser /etc/passwd; echo "did not find it, exit status $?"; echo "--- a pipeline reports only the last command ---"; false | true; echo "pipeline status $?"; set -o pipefail; false | true; echo "with pipefail set, status $?"
found root, exit status 0
did not find it, exit status 1
--- a pipeline reports only the last command ---
pipeline status 0
with pipefail set, status 1

Zero is success, which is backwards from most people’s instinct and is the convention everywhere: there is one way to succeed and many ways to fail, so failure gets the non-zero numbers.

grep -q is worth knowing on its own, it prints nothing and answers only with its exit status, which makes it the standard way to ask “does this file contain this” in a script.

And the last two lines are the important ones. false | true fails and then succeeds, and the pipeline reports 0, because a pipeline’s status is its last command’s status. A script checking that status sees success.

set -o pipefail changes it to report the first failure instead. On the same pipeline, the status becomes 1.

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

Here-documents

When a command wants input and you want to supply several lines of it inline:

# Debian 13 (trixie), x86_64
$ cat > note.txt <<EOF
first line
second line
EOF
echo "--- the here-doc wrote this ---"; cat note.txt; echo "--- and /dev/null discards ---"; ls /nosuchfile 2>/dev/null; echo "nothing above, exit status was $?"
--- the here-doc wrote this ---
first line
second line
--- and /dev/null discards ---
nothing above, exit status was 2

<<EOF feeds everything up to the next line reading EOF into the command. The word is arbitrary; EOF is convention. It is how a script writes a config file without a separate template.

<<'EOF' with the word quoted stops the shell expanding variables inside, which is what you want when the text contains $ characters that belong to something else. Unquoted, $HOME becomes your home directory; quoted, it stays $HOME.

/dev/null is the bin. Anything written to it disappears. 2>/dev/null silences errors you have decided you do not care about, and note the exit status was still 2, so discarding the message does not discard the failure. Silencing errors you have not read is how a broken script looks like a working one.

diff <(sort a.txt) <(sort b.txt)

Across distributions

Redirection and pipes are shell syntax, not commands, so they behave identically everywhere the shell is the same. The variation is which shell:

A script starting #!/bin/sh and using &> works on RHEL and fails on Debian, which is a genuinely common portability bug and one of the more annoying to diagnose, because the failure is a syntax error on a line that looks fine. Use #!/bin/bash if you are going to use bash features, and >file 2>&1 if you want the script to run anywhere.

Prove it

When output is not going where you expect:

# Is it on stdout or stderr? Throw one away and see what remains.
command 2>/dev/null      # only stdout survives
command 1>/dev/null      # only stderr survives

# Did the file get everything
command > out.txt 2>&1; wc -l out.txt

# Did the pipeline actually succeed
set -o pipefail
command | grep something; echo "status $?"

That first pair is the diagnostic worth remembering. Discard one channel and look at what is left, and you know which channel a given line came from, which is otherwise unanswerable, because on screen they are identical.

What trips people up

1. The error still appeared on screen

command > file redirects stdout only. Errors go to stderr and were never included.

command > file 2>&1, or command &> file in bash. And check the order: 2>&1 must come after the redirection that moves stdout.

2. > ate the file

It truncates before writing, immediately, with no prompt. >> appends.

The dangerous shape is sort file > file, which empties the file before sort reads it and leaves you with nothing. Use sort file > tmp && mv tmp file, or sort -o file file, which the tool supports precisely because of this.

Bash’s set -o noclobber makes > refuse to overwrite an existing file, with >| to override. Worth having on interactively.

3. Silencing errors and losing the failure

2>/dev/null hides the message. It does not change the exit status and it does not make the problem go away. A script that discards errors it has not read will carry on after something important failed.

Discard errors you have identified and decided to ignore. Never as a default.

4. A pipeline reports success when it failed

The status is the last command’s. curl badurl | grep pattern reports whatever grep did, and curl failing is invisible.

set -o pipefail, or check ${PIPESTATUS[@]}, which holds every command’s status individually.

5. Expecting a pipe to work for a command that wants a filename

diff and tar -f and most -f options want a name, not a stream. A pipe gives them nothing usable.

- means stdin for many of them (tar -xf -). Otherwise bash’s <(command) supplies a filename, as in the deeper panel above.

Work it through

A nightly script backs up a database and reports success every morning. Somebody notices the backup file is 0 bytes and has been for three weeks.

#!/bin/sh
pg_dump mydb | gzip > /backups/db.sql.gz 2>/dev/null
if [ $? -eq 0 ]; then
    echo "backup ok" | mail -s "nightly backup" ops@example.com
fi

Reason it out before reading on. There are three separate faults and they compound.

Fault one: the pipeline’s status is gzip’s. pg_dump can fail completely and gzip will still succeed, because compressing nothing is a perfectly valid thing to do, it produces a small, valid, empty archive. $? is 0, the test passes, the mail goes out.

Fault two: 2>/dev/null discarded the explanation. pg_dump almost certainly printed something useful (authentication failed, no such database, connection refused) every night for three weeks, and it went to the bin. This is what makes it three weeks rather than one morning.

Fault three: 2> is attached to the wrong thing. In a | b > file 2>/dev/null the redirection applies to gzip, the last command. pg_dump’s stderr was never touched by it and went to wherever cron sends it, which is usually mail to the account, which nobody reads. So the message was not even discarded where the author thought.

The fix, and each line earns its place:

#!/bin/bash
set -euo pipefail

if pg_dump mydb | gzip > /backups/db.sql.gz; then
    printf 'backup ok, %s bytes\n' "$(stat -c %s /backups/db.sql.gz)" \
      | mail -s "nightly backup" ops@example.com
else
    echo "backup FAILED" | mail -s "nightly backup FAILED" ops@example.com
fi

pipefail makes pg_dump’s failure the pipeline’s failure. Nothing is silenced, so errors reach cron’s mail. The if tests the pipeline directly rather than $? afterwards. And the success message reports the size, so a zero-byte backup announces itself instead of being reported as fine.

Now the question worth sitting with: why did three weeks pass? Not because the failure was subtle: pg_dump was shouting every night. It passed because the script was written to report success, and it did that faithfully. A check that can only say “ok” is not a check.

The habit: make the success message carry a number. “Backup ok” is unfalsifiable. “Backup ok, 2,481,003 bytes” is the same message with a fact in it, and a human reading “backup ok, 20 bytes” notices on the first morning.

Try it

Optional, on any machine.

1. ls /etc /nosuchdir > out.txt 2> err.txt, then look at both files. Say which channel each line came from.
2. Run the same command with 2>/dev/null, then with 1>/dev/null. Compare.
3. echo one > f.txt, echo two > f.txt, echo three >> f.txt, cat f.txt.
4. ls /etc | wc -l, then ls /etc | head -3. Notice how fast the second is.
5. grep -q root /etc/passwd; echo $? and then with a name that does not exist.
6. false | true; echo $?, then set -o pipefail and run it again.
7. diff <(sort /etc/passwd) <(sort /etc/passwd) and confirm it prints nothing.

Verification step. You have it when you can look at cmd 2>&1 > file and say, without running it, which channel ends up where.

Check yourself

References

• bash(1) - Linux man-pages project. Accessed 2026-08-07.
• stdin(3) - Linux man-pages project. Accessed 2026-08-07.
• tee(1) - Linux man-pages project. Accessed 2026-08-07.
• pipe(7) - Linux man-pages project. Accessed 2026-08-07.
• Shell Command Language - The Open Group. Accessed 2026-08-07.

Every block above with a distribution and architecture header was captured by running the command on a Debian 13 (trixie) container. Blocks without one are illustrative.