Process and service failures

Before you read. The monitoring says the order API is down. You SSH in and run systemctl is-active order-api. It prints active. You run it again. Still active. The application is definitely down, and the init system is definitely telling you it is up.

Neither of you is lying. systemd is answering a narrower question than the one you asked.

This lesson is about the distance between “a process exists” and “the service works”, because most service troubleshooting is spent in that gap. The tools will tell you which side of it you are on, but only if you read past the first line of output.

Some words you will need

unit

Anything systemd manages. A service is one kind of unit.

ActiveState

systemd's high-level view: active, inactive, failed, activating, deactivating.

SubState

The detail underneath it: running, exited, dead, auto-restart, start-pre.

main PID

The process systemd considers to be the service. Its exit ends the service.

exit code

The number a process returns. 0 is success; systemd reserves 200 and up for its own failures.

restart loop

A service that keeps failing and being restarted by its Restart= policy.

start limit

How many restarts in a window systemd will tolerate before giving up.

core dump

A snapshot of a crashed process's memory, written for later analysis.

What breaks without this

You trust a green check that means nothing. A dashboard polling systemctl is-active reports healthy for a service that has never once successfully started.

You restart instead of diagnosing. Restarting clears the symptom often enough to feel like a fix, so the actual cause is never found and the outage returns.

The error is on screen and gets skipped. systemd usually prints the exact reason in status, four lines down, and people read the first line and stop.

A dependency failure is chased in the wrong service. The unit that failed is frequently not the unit with the problem.

A crash loop is mistaken for a healthy service, because between restarts it genuinely is running.

The exit code usually is the answer

Start with a unit that cannot possibly work, because seeing the clean case makes the messy ones readable. This one points at a binary that does not exist:

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ sudo tee /etc/systemd/system/billing.service >/dev/null <<EOF
[Unit]
Description=Billing exporter

[Service]
Type=simple
ExecStart=/usr/local/bin/billing-export
Restart=on-failure
RestartSec=2
EOF
sudo systemctl daemon-reload; sudo systemctl start billing.service 2>&1; echo "--- what did that do ---"; systemctl is-active billing.service
--- what did that do ---
active

systemctl start printed no error and is-active says active. The binary does not exist. Nothing has ever run. This is the whole problem in five lines.

status tells the truth, and the useful part is not at the top:

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ systemctl status billing.service --no-pager 2>&1 | head -12
● billing.service - Billing exporter
     Loaded: loaded (/etc/systemd/system/billing.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: activating (auto-restart) (Result: exit-code) since Sat 2026-08-08 22:13:05 CDT; 274ms ago
 Invocation: c635874ff7924afb879862cdbe99cda1
    Process: 584700 ExecStart=/usr/local/bin/billing-export (code=exited, status=203/EXEC)
   Main PID: 584700 (code=exited, status=203/EXEC)
   Mem peak: 1.2M
        CPU: 2ms

Aug 08 22:13:05 localhost.localdomain systemd[1]: billing.service: Main process exited, code=exited, status=203/EXEC

Read it in this order, because the first line is the least informative:

status=203/EXEC is the answer. systemd reserves exit codes from 200 up for its own failures, and it names them:

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- the exit code is the whole diagnosis ---"; systemd-analyze exit-status 203 2>&1 | head -8
--- the exit code is the whole diagnosis ---
NAME STATUS CLASS
EXEC    203 systemd

EXEC means systemd could not execute the command at all. The binary is missing, is not executable, or its interpreter line points at nothing.

The systemd exit codes worth recognising on sight:

The distinction that matters: a code in the 200s means the service never started, so the problem is in the unit file or the filesystem, and the application’s own logs will be empty. A code of 1 means the application ran and failed on its own terms, so its logs are exactly where to look. Knowing which of those you have saves a great deal of time.

The restart loop and where it ends

Restart=on-failure is in that unit, which is why Active: said activating (auto-restart) rather than failed.

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ sleep 20; echo "--- twenty seconds later, the loop has given up ---"; systemctl is-active billing.service; systemctl status billing.service --no-pager 2>&1 | sed -n "5p;7p"; echo "--- and the journal says why it stopped trying ---"; journalctl -u billing.service -n 3 --no-pager -o cat
--- twenty seconds later, the loop has given up ---
activating
     Active: activating (auto-restart) (Result: exit-code) since Sat 2026-08-08 22:13:40 CDT; 1s ago
    Process: 585562 ExecStart=/usr/local/bin/billing-export (code=exited, status=203/EXEC)
--- and the journal says why it stopped trying ---
billing.service: Failed at step EXEC spawning /usr/local/bin/billing-export: No such file or directory
billing.service: Main process exited, code=exited, status=203/EXEC
billing.service: Failed with result 'exit-code'.

Twenty seconds and hundreds of failed starts later, is-active still says activating. Not failed. A monitoring check that treats anything other than the literal string failed as healthy will never fire on this.

And the first journal line is the plain-English version of 203: Failed at step EXEC spawning /usr/local/bin/billing-export: No such file or directory. That is the whole investigation, and it was one command away the entire time.

A restart loop does eventually stop. StartLimitBurst= (5 by default) within StartLimitIntervalSec= (10 seconds) is the budget; exceed it and the unit enters failed with “start request repeated too quickly”, and systemd will not try again until you systemctl reset-failed. Whether you see that or a loop that continues depends on how RestartSec= compares to the interval: a two-second delay spaces the attempts widely enough that the burst counter keeps resetting, which is exactly what happened above.

Which is worth knowing as a design point, not just a diagnostic one. A service that retries slowly enough never trips its own limit, so it will loop until somebody notices. That may be what you want for a service waiting on a database, and it is not what you want for one with a typo in its unit file.

Active is not the same as working

Here is the version of the problem that costs real time: the unit is genuinely running, systemd is genuinely correct, and the service is genuinely down.

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ sudo systemctl stop billing.service >/dev/null 2>&1; sudo rm -f /etc/systemd/system/billing.service; sudo tee /etc/systemd/system/api.service >/dev/null <<EOF
[Unit]
Description=Order API

[Service]
ExecStart=/bin/sh -c "echo starting the order API; exec sleep 900"
EOF
sudo systemctl daemon-reload; sudo systemctl start api.service; sleep 1; echo "--- systemd is perfectly happy ---"; systemctl show api.service -p ActiveState -p SubState -p MainPID; echo "--- and nothing is serving port 9090 ---"; ss -ltn "sport = :9090"
--- systemd is perfectly happy ---
ActiveState=active
SubState=running
MainPID=586422
--- and nothing is serving port 9090 ---
State Recv-Q Send-Q Local Address:Port Peer Address:Port

active, running, a real main PID, and nothing listening. ss printed its header and no rows, which is what “no socket matches” looks like.

There is no bug here. A process is running, exactly as systemd reports. It is simply not doing the job the service exists to do, and the init system has no way to know that. In the real world this is a daemon that started, failed to read its config, logged a complaint, and sat there; or one that crashed a worker thread while its supervisor kept breathing.

So the diagnostic move is to stop asking systemd and ask the service:

ss -ltnp is the single most valuable of those for a network service, because “is it listening” is the closest thing to a binary answer you will get, and if it is listening on 127.0.0.1 when it should be on 0.0.0.0 you have found the bug outright.

cat /proc/<pid>/wchan; echo        # the kernel function it is sleeping in
sudo cat /proc/<pid>/stack         # kernel stack, if available
ps -o pid,stat,wchan:30,etime,time,cmd -p <pid>

When a process is killed by a signal

The other way a service dies is that something kills it, and the exit status records which signal did it.

# AlmaLinux 10.2, aarch64
$ echo "--- a process killed by a signal, and what the shell reports ---"; sh -c "kill -SEGV \$\$"; echo "exit status: $?"; echo "--- which signal is 11 ---"; kill -l 11
--- a process killed by a signal, and what the shell reports ---
/bin/sh: line 1:     2 Segmentation fault      (core dumped) sh -c "kill -SEGV \$\$"
exit status: 139
--- which signal is 11 ---
SEGV

139 is 128 plus 11, and 11 is SIGSEGV. That arithmetic is the convention: a status above 128 means the process was killed by signal (status minus 128).

137 deserves special attention because it looks like a crash and usually is not: it is most often the kernel’s out-of-memory killer choosing your process. journalctl -k | grep -i 'killed process' confirms it in one command, and the fix is a memory problem, not an application bug.

In systemctl status this appears as code=killed, signal=SEGV rather than code=exited, and that distinction is worth reading carefully: exited means the program chose its fate, killed means something else chose it.

coredumpctl list                     # every crash the system has recorded
coredumpctl info 12345               # signal, command line, and a stack trace if symbols allow
coredumpctl debug 12345              # open it in gdb
coredumpctl dump 12345 > core.dump   # extract the raw dump

Dependencies, and the unit that is not the problem

A failing unit is often collateral damage.

systemctl list-dependencies app.service          # what it needs
systemctl list-dependencies --reverse app.service # what needs it
systemctl --failed                                # everything currently failed
journalctl -b -p err                              # this boot, errors and worse

systemctl --failed first, always. If three units are failed, they are probably not three problems: they are one problem and two consequences. Fix the one whose failure is earliest in the journal.

The ordering directives that produce this:

After= does not require anything and Requires= does not order anything. That is the single most common misunderstanding in unit files. Requires=db.service without After=db.service starts both at once and your service races the database. Nearly always you want both, and nearly always the pair is what the vendor’s unit already has.

And a dependency being started is not a dependency being ready, which is the Type= problem from earlier viewed from the other side. With Type=simple on the database, After= waits for a fork and nothing more. The answer that holds up is for the application to retry its connection, exactly as lesson 61 argued for containers.

The order to work in

1. systemctl --failed to see the whole picture before fixating on one unit.
2. systemctl status <unit> and read down to Process: and the log lines. The answer is usually there.
3. Decode the exit code. 200s mean it never started, so look at the unit file. Anything else means it ran, so look at its logs.
4. journalctl -u <unit> -b --no-pager for the full story rather than the ten-line tail.
5. If it claims to be active, verify independently. ss -ltnp, a curl, a query. Do not accept the init system’s word for it.
6. Check what it depends on before assuming the fault is here.
7. systemctl cat <unit> to see the unit file and every drop-in that modifies it, which is where a surprising number of causes hide.

Across distributions

systemd is systemd, so almost everything here is portable. The differences are in what surrounds a failing unit rather than in the unit itself.

The “starts on install” row surprises people moving in either direction. Debian and Ubuntu policy is that installing a service package enables and starts it, so a package installed at 4pm is listening on a port at 4pm. The RHEL family installs it stopped and disabled, and waits for you. Neither is wrong, and assuming the wrong one gives you either a service you did not know was running or one you were certain you had installed.

The mandatory access control row matters here because a service that starts fine by hand and fails under systemd is the classic shape of a policy denial. Under systemd the unit runs in a confined domain that your interactive shell does not, so “it works when I run it myself” is evidence about the confinement rather than evidence that the unit file is wrong.

Prove it

The diagnosis is almost always in the first three commands, and the order matters because each one narrows what the next needs to explain:

# The failing command and its exit code, on the Process: line
systemctl status <unit> --no-pager -l

# Why, in the service's own words
journalctl -u <unit> -b --no-pager | tail -40

# Is it failed, or looping, or merely inactive
systemctl is-active <unit>; systemctl is-enabled <unit>
systemctl --failed

# What the unit is actually running, after every drop-in is applied
systemctl cat <unit>
systemctl show <unit> -p ExecStart -p User -p WorkingDirectory -p Restart

# For a process rather than a unit
ps -o pid,stat,wchan:20,etime,cmd -p <pid>

systemctl cat is the one to build a habit around. It prints the vendor unit followed by every drop-in in the order they apply, so it answers “what is this unit actually configured to do” rather than “what does the file I am looking at say”. Editing the right file and being overridden by a drop-in you forgot about is a genuinely common half hour.

What trips people up

1. Reading the first line of systemctl status and stopping

The coloured failed line tells you it failed, which you knew. The Process: line a few rows down carries the command that ran and the code it exited with, and that is the diagnosis. Everything else on the screen is context.

2. Treating active as working

With Type=simple, active means systemd forked the process successfully and nothing more. The service can be up, listening on nothing, and failing every request while systemd reports it green. Check the port with ss -ltnp and the service’s own log rather than trusting the state word.

3. Missing a restart loop because the unit says activating

A unit crashing and restarting on a timer never settles into failed, so it does not appear in systemctl --failed and a status check catches it mid-restart looking healthy. activating (auto-restart) is the tell, and journalctl -u <unit> shows the same startup repeating on a rhythm.

4. Reading exit codes as the application’s

Codes at 200 and above belong to systemd and mean the service never started: 203/EXEC for a binary that is missing or not executable, 200/CHDIR for a working directory that does not exist. Anything above 128 is a signal, minus 128, so 137 is SIGKILL and 143 is SIGTERM. Only the small numbers came from the program.

5. kill -9 on a D-state process

Uninterruptible sleep means the process is inside a system call waiting on I/O, and it cannot receive any signal until that completes, SIGKILL included. The signal is queued rather than ignored. Fix the I/O, which is usually a hung NFS mount or a failing disk, and read wchan to see what it is waiting on.

6. Confusing Requires= with After=

Requires= says the other unit must be there and stops this one if it fails. After= says only “start me later”. They are independent, so a unit with Requires= alone can be started before the thing it requires is ready, which produces an intermittent failure that looks like a race because it is one.

Work it through

A unit fails on start. systemctl status reports:

Active: failed (Result: exit-code) since Sat 2026-08-08 02:11:04 UTC
Process: 4181 ExecStart=/usr/local/bin/reportd --config /etc/reportd.yaml (code=exited, status=203/EXEC)

journalctl -u reportd has nothing from the application at all.

Reason it out before reading on.

Read the code rather than the word failed. 203 is in systemd’s range, which means systemd could not execute the command, so the application never ran. That immediately explains the empty journal: there was no process to write anything, and hunting through the application’s config would be wasted time.

EXEC covers a short list, and each item is one command. It is a small list, and each item is one command:

ls -l /usr/local/bin/reportd     # does it exist, is it executable
head -1 /usr/local/bin/reportd   # if a script, is the interpreter real
file /usr/local/bin/reportd      # right architecture, not a broken symlink

A missing execute bit, a shebang pointing at an interpreter that is not installed, and a dangling symlink all produce this same code.

If the file looks fine, ask who is being stopped from running it. The unit does not run as you:

systemctl show reportd -p User -p Group -p RootDirectory
sudo -u reportd /usr/local/bin/reportd --config /etc/reportd.yaml

Running it by hand as the service account either reproduces the failure with a better error, or succeeds and tells you the confinement is involved.

Running fine by hand points at the confinement. On the RHEL family that is one command, and a binary in /usr/local/bin is a common source of it because the default label there is not one service domains may execute:

sudo ausearch -m AVC -ts recent
ls -Z /usr/local/bin/reportd

The lesson underneath: an exit code told us which layer failed before any file was opened. Codes in systemd’s range mean the problem sits between systemd and the binary, so the application, its configuration, and its dependencies are all out of scope until that is resolved.

Try it

Optional, and a VM or container with systemd running is enough.

1. Write a trivial unit whose ExecStart points at a script that does not exist. Start it and read the exact code on the Process: line. Create the script without an execute bit and try again. Note that you get the same code from two different causes.
2. Give the script a shebang naming an interpreter that is not installed, such as #!/usr/bin/python4. Start it and confirm the code again, then confirm that the file itself is present and executable. This is the case that makes people doubt their own eyes.
3. Make the script exit 1 after five seconds and add Restart=always with RestartSec=2. Run systemctl status repeatedly and catch it reporting activating (auto-restart). Confirm it never appears in systemctl --failed.
4. Add StartLimitBurst=3 and StartLimitIntervalSec=60, then watch where the loop stops.

Verification step. You have step 3 right when you can explain why a monitoring check written against systemctl is-failed would report this service as healthy while it restarts every seven seconds indefinitely.

For the exam

systemctl status shows the failing command and its exit code in the Process: line. That is the diagnosis, not the first line.

Exit codes 200 and up are systemd’s, meaning the service never started. 203/EXEC is a missing or non-executable binary.

Exit status above 128 means killed by a signal, status minus 128. 137 is SIGKILL, 139 is SIGSEGV, 143 is SIGTERM. That is what a shell or a container runtime reports; systemctl status renders the same death as code=killed, signal=KILL, so do not go looking for the number there.

active does not mean working. With Type=simple it means systemd forked successfully, nothing more.

A unit in a restart loop reports activating (auto-restart), not failed.

systemctl --failed lists everything currently failed.

journalctl -u <unit> is where the reason lives.

Requires= is dependency, After= is ordering. They are independent and you usually need both.

A D-state process cannot be killed, not even with -9. It is blocked on I/O.

Where this sits

Lesson 33 wrote unit files and lesson 34 read the journal; this is what to do when a unit will not behave. Lesson 29’s signals explain the numbers above 128, and status 137 usually leads to lesson 75 and the OOM killer rather than to any bug in the service.

The next lesson goes one layer lower, to the point where the logs start naming hardware.

References

• systemctl(1) - freedesktop.org. Accessed 2026-08-09.
• systemd.service(5) - freedesktop.org. Accessed 2026-08-09.
• systemd.exec(5), process exit codes - freedesktop.org. Accessed 2026-08-09.
• signal(7) - man7.org. Accessed 2026-08-09.

The commands here were run on a real machine, not written from memory. The systemd transcripts come from Fedora CoreOS 44.20260707.3.1 on aarch64, where systemd is genuinely PID 1, so the restart loop, the journal, and systemd-analyze are the real ones rather than a container approximation. The billing.service unit really did point at a binary that does not exist, and is-active really did report active while it had never once run. The segfault transcript is from AlmaLinux 10.2 on aarch64, run natively so the shell’s own report of the signal is the one you would see.