How to troubleshoot

Before you read. A ticket says “the reporting tool is broken”. That is all it says. The machine is up, you can log in, and nobody can tell you what “broken” means.

You have to start somewhere, and where you start decides whether this takes five minutes or the rest of the afternoon.

Most people learn troubleshooting by absorbing it, which works eventually and takes years. There is a method underneath, and it is small enough to state on one page: turn the symptom into a guess you can test, design a test that can only come out one way, change one thing, and write down what you did.

The exam asks about this because it is the skill the other twelve lessons in this block are in service of. Knowing that df -i exists is worth nothing if you never think to suspect the filesystem.

Some words you will need

symptom

What somebody noticed. Almost never the fault itself.

hypothesis

A specific guess about the cause, phrased so it could be wrong.

discriminating test

An observation whose result rules something in or out. The whole game.

bisection

Halving the space of possible causes with each test.

blast radius

What a change can affect if it goes wrong.

escalation

Handing the problem to somebody with more access or more knowledge.

What breaks without this

Fixes get applied at random. Three changes go in at once, the symptom moves, and now nobody knows which change did what.

The wrong layer gets blamed. Hours go into the application because the error mentioned the application, when the disk was full.

The same outage happens twice. Nothing was written down, so the next person starts from nothing, including you in six months.

Nobody escalates in time. An engineer spends four hours on something the storage team would have recognised in four minutes.

Changes make it worse. A speculative fix breaks something that was working, and now there are two faults.

Symptom, hypothesis, test

The loop is three steps and you repeat it until the cause is cornered.

A symptom is what somebody reports: “the reporting tool is broken”, “the site is slow”, “I cannot log in”. It is a description of an experience, and it is usually not a description of a fault. The first job is to convert it into something specific enough to be wrong about.

That means asking what “broken” means in observable terms. Broken how? For everybody or one person? Since when? What is the exact error?

Hypothesis comes next, and the phrasing matters more than people expect. “It is a permissions problem” is not a hypothesis, because no observation could disprove it. “The service cannot write to /var/lib/reports because the directory is owned by root and the service runs as reports” is a hypothesis: one ls -ld settles it.

Then the test. A good test is one whose outcome you cannot predict, and where either outcome tells you something. If you already know what it will say, you are not testing, you are reassuring yourself.

Restarting deserves its own warning, because it is the most common first move and it destroys evidence. It often clears the symptom, which feels like success and teaches you nothing, and the fault returns on Sunday when you are not there. Collect state first, restart second.

Read the error you were given

Error messages are read carelessly, and the ones that mislead do so in a consistent way: they name the thing the tool was working on, not the thing that was missing.

Here is a script that exists, is executable, and will not run.

# AlmaLinux 10.2, aarch64
$ echo "--- the script is right there, and executable ---"; ls -l report.sh; echo "--- so what does running it say ---"; ./report.sh; echo "exit status: $?"
--- the script is right there, and executable ---
-rwxr-xr-x. 1 root root 36 Aug  9 04:06 report.sh
--- so what does running it say ---
/bin/sh: line 1: ./report.sh: cannot execute: required file not found
exit status: 127

Look carefully at what that says. ./report.sh: cannot execute: required file not found. The path it names is ./report.sh, which ls just proved is there.

The message is accurate and the reading is wrong. The file that was not found is a different one, and the message does not name it. When you run a script, the kernel reads the first line, finds the interpreter, and executes that. If the interpreter is missing, the exec fails, and the shell reports the failure against the file you asked for.

You can watch the kernel say so:

# AlmaLinux 10.2, aarch64
$ echo "--- ask the kernel which file it could not find ---"; strace -f -e trace=execve ./report.sh 2>&1 | head -4
--- ask the kernel which file it could not find ---
execve("./report.sh", ["./report.sh"], 0xfffffa3f0a88 /* 8 vars */) = -1 ENOENT (No such file or directory)
strace: exec: No such file or directory
+++ exited with 1 +++

ENOENT on execve of a file that exists is the signature. The kernel resolved the interpreter, failed to find it, and returned the error against the original call.

# AlmaLinux 10.2, aarch64
$ echo "--- the file the kernel wanted is named on line 1 ---"; head -1 report.sh; echo "--- does it exist ---"; ls -l /bin/bahs 2>&1; echo "--- fix the typo, run it again ---"; sed -i "1s|.*|#!/bin/bash|" report.sh; ./report.sh; echo "exit status: $?"
--- the file the kernel wanted is named on line 1 ---
#!/bin/bahs
--- does it exist ---
ls: cannot access '/bin/bahs': No such file or directory
--- fix the typo, run it again ---
report generated
exit status: 0

#!/bin/bahs. One transposition, and the error pointed at the wrong file the whole time.

How to read an error message properly, in the order that pays:

• Read every word, including the ones that look like boilerplate. cannot execute and not found are two different claims and the combination is the clue.
• Ask which file, which line, which user, which host. If the message does not say, that absence is information.
• Distinguish what the tool observed from what the tool concluded. Tools report conclusions confidently and are often reasoning from one layer too high.
• Search the exact string, in quotes, before searching your paraphrase of it.
• If the message is thin, ask something lower down. strace for syscalls, ltrace for library calls, -v or --debug for the tool’s own view.

Exit status 127 is worth memorising: command not found, or interpreter not found. 126 means found but not executable, which is usually a mode bit or a noexec mount. Those two are frequently mistaken for each other.

What changed

Machines do not spontaneously break. Something changed, and the question is what.

The awkward part is that the change is often not on the machine. A certificate expired, a DNS record was updated, a firewall rule was added upstream, a dependency published a new version, or a disk crossed a threshold that had been climbing for a month. “Nothing changed” from a colleague means “I did not change anything”, which is a much smaller claim.

Package history is the easiest thing to check and is frequently decisive:

# AlmaLinux 10.2, aarch64
$ echo "--- what changed on this machine, most recent first ---"; rpm -qa --last | head -5
--- what changed on this machine, most recent first ---
krb5-libs-1.21.3-10.el10_2.aarch64            Tue Jun  2 11:13:49 2026
rootfiles-8.1-54.el10.noarch                  Tue Jun  2 11:12:21 2026
less-661-3.el10.aarch64                       Tue Jun  2 11:12:21 2026
yum-4.20.0-22.el10_2.alma.1.noarch            Tue Jun  2 11:12:20 2026
xz-5.6.2-4.el10_0.aarch64                     Tue Jun  2 11:12:20 2026

Everything installed on the same day, which is what a container image looks like. On a real server this list is a timeline, and a package updated an hour before the incident is a strong lead.

The places worth checking, roughly in order of how often they pay:

rpm -Va deserves more use than it gets. It verifies every installed file against the package’s recorded checksum, mode, and owner, so it finds the config somebody edited by hand two years ago and forgot. On Debian the equivalent is debsums -c.

systemctl list-timers --all          # what is scheduled and when it next runs
sudo find /etc -name '*.crt' -o -name '*.pem' | \
  xargs -I{} sh -c 'echo -n "{} "; openssl x509 -enddate -noout -in {} 2>/dev/null'

One thing at a time

This is the rule people know and break, and the reason it matters is not tidiness.

Change three things, and if the symptom clears you have learned nothing about which mattered. Worse, if one of your three changes introduced a new fault while another fixed the original, the symptom persists and you now have two problems tangled together with no way to separate them.

So: one change, then observe, then keep or revert. Write down each one as you go, because after ninety minutes you will not remember whether you reverted the sysctl or only meant to.

Before any change, ask three questions. How do I undo this? What else could it affect? Is this reversible at all? A chmod -R on the wrong directory, a truncate on the wrong file, and a dd with the arguments swapped are not reversible, and no amount of care afterwards helps.

Keep the original when you edit config:

sudo cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.$(date +%F)

Then validate before restarting, because most services will tell you whether they can parse the file: nginx -t, sshd -t, visudo -c, named-checkconf. Restarting into a broken config on a remote machine is a memorable way to end an afternoon.

Across distributions

The method is identical everywhere, because it is a way of thinking rather than a toolset. What moves is where the evidence is kept.

The journal is the portable answer, which is the practical reason to reach for journalctl -u before hunting for a file. It is the same command, with the same options, on every systemd machine, and it does not care what the distribution decided to call /var/log/messages.

The rpm -V row is the one that repays knowing. On the RHEL family you can ask “has anything about this package changed since it was installed” and get a per-file answer covering mode, ownership, and checksum, which turns “what changed” from an interview question into a command. The Debian equivalent needs debsums installed ahead of time, and on a machine you have just been handed it will not be.

Prove it

Collect before you change. This is the whole of it, and it takes under a minute:

# The exact error, from the machine rather than from the person reporting it
journalctl -u <unit> -p err -b --no-pager | tail -20

# What changed, and when
rpm -qa --last | head            # Debian: tail /var/log/dpkg.log
systemctl list-units --failed
uptime                           # has it restarted since it last worked

# The four cheap eliminations, before any theory at all
df -h; df -i                     # space, and inodes separately
free -h
ss -ltnp | head                  # is it even listening
id <the service account>         # is it running as who you think

Run this before restarting anything. A restart frequently clears the symptom and always destroys the state that would have explained it, so the fault comes back on a Sunday with nothing left to read. Ninety seconds of collection buys you the ability to diagnose it the second time.

What trips people up

1. Restarting first

It is the most common opening move and it is evidence destruction. The process table, the open file handles, the memory usage, and anything the service was about to log all go away. If the restart works you have learned nothing, and you will be back.

2. “It is a permissions problem”

That is a hunch, not a hypothesis, because no observation could disprove it. A hypothesis names a specific thing that a single command settles: the service runs as reports and /var/lib/reports is owned by root, so ls -ld decides it in one line.

3. Changing two things at once

Then it works and you cannot say why, so you cannot write it down, tell anybody, or recognise it next time. Worse, one of the two changes may have introduced a fault that surfaces next week looking unrelated.

4. Treating the reporter’s diagnosis as the symptom

“The database is down” is almost always someone’s conclusion rather than their observation. What they saw was an error in an application. Ask what they typed and what appeared, because the conclusion has already thrown away the useful part.

5. Reading the last error instead of the first

A failure cascades, so the end of the log is full of consequences. The first error in the sequence is the one with the cause in it, and everything after it is noise generated by the thing that already went wrong.

6. Never asking what changed

Systems that ran for a year do not spontaneously develop faults. Something changed: a package, a certificate expiring on its own schedule, a disk filling, a group membership. “What changed” is the highest-yield question in the topic and it is the one most often skipped, because the answer is usually “nothing” and that answer is usually wrong.

Work it through

Putting the pieces together on the fault above, as it would actually go.

The report says the reporting tool is broken. First question: what does broken mean? The answer is that running it prints an error. Ask for the exact text and you get ./report.sh: cannot execute: required file not found.

That message suggests a missing file, so the first hypothesis is the obvious one: the script is gone. ls -l report.sh shows it present and executable. Hypothesis dead in one command, which is exactly what a good test does.

Second hypothesis: the message means a different file. That gives a discriminating test, because if some other file is missing, the kernel knows which. strace -e trace=execve shows ENOENT against a path that exists, which only happens when the interpreter cannot be resolved.

Third: the interpreter named in the script is wrong. head -1 shows #!/bin/bahs, and ls /bin/bahs confirms it does not exist. One change, one character, and it runs.

Four commands, no restart, and the fix was the last thing done rather than the first. Note that the middle step is what beginners skip: when the obvious reading of an error turned out to be wrong, the next move was to ask a lower layer rather than to guess again.

Try it

Optional, and worth doing on a VM you can restore, because the point is to break something on purpose and practise the loop rather than the fix.

1. Break a service in a way you will forget. Pick one that is running, then do exactly one of these without writing it down: rename its binary, change the user in its unit file, revoke read on its config, or fill the filesystem its working directory sits on. Walk away for ten minutes.
2. Come back and treat it as a report. Write the symptom in observable terms before typing anything, then run the collection block from Prove it and read it before forming a theory.
3. Write your hypothesis down as a sentence containing a path, a user, or a number. If you cannot, it is still a hunch.
4. Test it with one command whose answer you cannot predict, then fix exactly one thing.

Verification step. You have done this properly when you can state, without looking, which single command eliminated the largest number of candidate causes. If your answer is “the one that fixed it”, the diagnosis was luck, so restore the snapshot and run it again with a different fault.

Do it a second time with two faults introduced at once. That version teaches the lesson about changing one thing at a time faster than reading about it does.

For the exam

Symptom, hypothesis, test. Convert a report into something specific enough to be wrong about.

A good test discriminates. If you know the answer already, it is not a test.

Change one thing at a time, and record each change.

Establish what changed before assuming the machine changed itself.

Read the exact error text. It may name the wrong file, and 127 means “command or interpreter not found”.

Collect evidence before restarting. Restarting destroys the state that would have explained it.

Have a way back in before touching SSH, firewall, or network configuration.

Escalate on a clock, on access limits, or on blast radius.

Where this sits

This lesson is the method; the rest of block F is the specific knowledge it operates on. Lesson 65 is where the evidence lives, and the lessons after it work through the layers in turn: boot, filesystems, disk space, services, hardware, network, permissions, and performance.

The next lesson is about noticing a fault before a customer does, which is the only thing better than diagnosing one quickly.

References

• strace(1) - man7.org. Accessed 2026-08-09.
• execve(2), including ENOENT on a missing interpreter - man7.org. Accessed 2026-08-09.
• rpm(8) - man7.org. Accessed 2026-08-09.
• dnf history - DNF. Accessed 2026-08-09. If the fault is live and people are asking. This topic is a method for finding a cause. The hour after it breaks on the Network+ track is everything happening around you while you use it: who runs the incident, what to say before you know anything, and why a postmortem that names a person has found nothing. It is written for a network fault and none of it is about networks.

The commands here were run on a real machine, not written from memory. The transcripts come from AlmaLinux 10.2 on aarch64, run natively so the shell’s own error text is the one you would see. report.sh really did carry #!/bin/bahs, and the error really did name the script rather than the interpreter. The package list is from a container image, which is why every package shares an install date; on a server that column is a timeline.