Boot failures and recovery

Before you read. The machine was rebooted for a routine kernel update. It has not come back. The console shows a wall of text, ending in a prompt that is not a login prompt, and it wants a root password you may not have.

Everything you normally use to investigate a Linux machine assumes the machine booted.

Boot failures feel worse than other faults because your tools are gone with the system. There is no SSH, no journal you can query from a comfortable terminal, and frequently no shell.

The good news is that boot is a strict sequence, so the screen tells you roughly where it stopped, and where it stopped narrows the cause sharply. Lesson 09 described that sequence working. This is the same ground when it does not.

Some words you will need

firmware

UEFI or BIOS. Runs before anything of yours, and picks what to load.

bootloader

GRUB on most systems. Loads a kernel and hands it parameters.

initramfs

A small temporary root filesystem holding the drivers needed to find the real one.

kernel command line

Parameters the bootloader passes. Includes which device holds the root filesystem.

target

A systemd state to reach. multi-user.target is a normal server.

rescue

Single user with the local filesystems mounted.

emergency

A shell with almost nothing started and root mounted read-only.

chroot

Running commands with a different directory as the root, so tools act on the broken system.

What breaks without this

A production machine stays down while somebody works out how to get a prompt.

The rebuild is chosen over the repair. Reinstalling is a way to avoid diagnosis, and it destroys whatever local state had not been backed up.

The recovery makes it worse. Reinstalling a bootloader from the wrong root or against the wrong disk is a common way to lose the other operating system on the machine.

Nobody knows where the console is. The fix takes two minutes and access to it takes two hours.

How far did it get?

Work out which stage failed before touching anything, because each stage has different tools.

The most useful distinction is whether you got a GRUB menu. Before it, the problem is firmware, disk, or bootloader. After it, the kernel is running and you have parameters you can edit, which is a much better position.

The kernel command line

Everything after GRUB depends on the parameters it passed. This is what a healthy one looks like:

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- what the kernel was told at boot ---"; cat /proc/cmdline; echo "--- where the system is trying to get to, and where it is ---"; systemctl get-default; systemctl is-system-running
--- what the kernel was told at boot ---
BOOT_IMAGE=(hd0,gpt3)/boot/ostree/fedora-coreos-b03a2111206afe90841d87f839937afc8dce4e7fa9e11e5ac0c314f03439c33f/vmlinuz-7.1.3-200.fc44.aarch64 rw ostree.prepare-root.composefs=0 ostree=/ostree/boot.1/fedora-coreos/b03a2111206afe90841d87f839937afc8dce4e7fa9e11e5ac0c314f03439c33f/0 ignition.platform.id=applehv console=tty0 console=hvc0 root=UUID=159ca8aa-2ced-4891-90cc-0bde469c40f8 rw rootflags=prjquota boot=UUID=ec4a7bbc-82cf-4c92-a421-381dd40c0e0c
--- where the system is trying to get to, and where it is ---
multi-user.target
degraded

That line is long because this is an ostree system, and most of it is distribution machinery. The parts that matter on any system are the same three:

• BOOT_IMAGE= which kernel was loaded, and from which partition.
• root=UUID=159ca8aa-... where the root filesystem is. This is the parameter that causes the most boot failures, because a UUID that no longer exists produces a kernel that boots perfectly and then has nowhere to go.
• console=tty0 console=hvc0 where kernel messages are sent. Worth knowing when the screen is blank but the machine seems alive: the messages may be going to a serial console you are not watching.

Editing it from the boot menu is the single most valuable recovery skill, and it requires no media and no preparation:

1. At the GRUB menu, highlight the entry and press e.
2. Find the line beginning linux or linuxefi.
3. Edit it. Append systemd.unit=rescue.target, or single, or init=/bin/bash.
4. Ctrl-X or F10 to boot with the change.

Nothing is saved. A reboot restores the original entry, which makes this safe to experiment with.

What to append, and what each gives you:

init=/bin/bash is the one to remember for a lost root password, since it bypasses the password prompt that rescue mode would give you. Root is mounted read-only, so mount -o remount,rw / comes first, and on a system with SELinux a password change needs touch /.autorelabel before rebooting or the next boot fails for a different reason.

lsinitrd /boot/initramfs-$(uname -r).img | head -40    # contents
lsinitrd /boot/initramfs-$(uname -r).img | grep -iE 'virtio|nvme|megaraid'

sudo dracut -f                                  # current kernel, in place
sudo dracut -f --kver 6.1.0-30                  # a specific kernel
sudo dracut -f --add-drivers "virtio_blk virtio_scsi"
sudo dracut -f --regenerate-all                 # every installed kernel

mount -o remount,rw /sysroot
chroot /sysroot
# fix the problem
exit
mount -o remount,ro /sysroot
exit

Rescue and emergency

Both give you a shell on a system that will not boot normally, and the difference is how much has been started.

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- the targets that exist for recovery ---"; systemctl list-units --type=target --all --no-pager 2>/dev/null | grep -E "rescue|emergency|multi-user|graphical" | head -5
--- the targets that exist for recovery ---
  emergency.target                   loaded    inactive dead   Emergency Mode
  multi-user.target                  loaded    active   active Multi-User System
  rescue.target                      loaded    inactive dead   Rescue Mode

Rescue mounts the local filesystems and starts a minimal set of services, so the system is broadly usable and single user. Reach for it when the machine mostly works and one service or one configuration file is the problem.

Emergency starts almost nothing. Root is mounted read-only, other filesystems are not mounted at all, and there is no networking. It is the right choice when the filesystems themselves are suspect, because it touches as little as possible.

Both prompt for the root password, which is the practical catch. A system with no root password set, which is increasingly common on cloud images, cannot offer either, and init=/bin/bash is the way in.

You can also reach them from a running system, which is worth knowing for maintenance rather than recovery:

sudo systemctl rescue          # drop to rescue now
sudo systemctl emergency       # drop to emergency now
sudo systemctl default         # back to normal

console=tty0 console=ttyS0,115200n8       # kernel command line: both screen and serial

Degraded means it arrived with casualties

A boot can succeed and still be wrong, and systemd has a specific word for it. Look again at the first capture: systemctl is-system-running reported degraded, not running.

That means the machine reached multi-user.target and something failed on the way. It is easy to miss, because a degraded system gives you a login prompt and looks fine.

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- degraded means it got there with casualties. which ones ---"; systemctl --failed --no-pager
--- degraded means it got there with casualties. which ones ---
  UNIT                LOAD   ACTIVE SUB    DESCRIPTION
● rpm-ostreed.service loaded failed failed rpm-ostree System Management Daemon

Legend: LOAD   → Reflects whether the unit definition was properly loaded.
        ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
        SUB    → The low-level unit activation state, values depend on unit type.

1 loaded units listed.

One unit, named, with its load and active states. From here it is lesson 69’s territory: systemctl status rpm-ostreed.service and the journal for that unit.

Make is-system-running a habit after any reboot. It returns running, degraded, maintenance, or starting, and it is a single word that tells you whether to look further. A monitoring check on it costs nothing and catches the service that quietly failed to start three reboots ago.

Boot timing is worth the same glance, because a boot that succeeds slowly is usually a boot that timed out waiting for something:

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- how long the last boot took, and which units were slowest ---"; systemd-analyze; systemd-analyze blame --no-pager 2>/dev/null | head -5
--- how long the last boot took, and which units were slowest ---
Startup finished in 316ms (kernel) + 2.108s (initrd) + 3.179s (userspace) = 5.604s 
multi-user.target reached after 3.086s in userspace.
2.210s dev-hvc4.device
2.210s sys-devices-virtual-tty-hvc4.device
2.208s sys-devices-virtual-tty-hvc1.device
2.208s dev-hvc1.device
2.208s dev-hvc0.device

Three phases, separately timed: 316 ms in the kernel, 2.1 seconds in the initramfs, 3.2 seconds in userspace. That split localises a slow boot immediately, because a long initrd phase and a long userspace phase have nothing in common.

blame sorts units by how long each took, and it is frequently misinterpreted. A unit at the top of the list is not necessarily delaying the boot: it may have started early and run alongside everything else. systemd-analyze critical-chain shows the dependency path that actually determined the total, which is the one to act on.

A unit taking a suspiciously round number, 90 seconds especially, is almost always a timeout rather than work. 90 seconds is systemd’s default device timeout, so it usually means something waited for a device that never appeared, which brings you back to fstab and lesson 67’s nofail.

# 1. identify the partitions
lsblk -f

# 2. mount the real root somewhere
sudo mount /dev/sda2 /mnt

# 3. mount the other filesystems the system expects
sudo mount /dev/sda1 /mnt/boot
sudo mount /dev/sda1 /mnt/boot/efi        # if EFI is separate

# 4. give the chroot the kernel interfaces it needs
for d in dev proc sys run; do sudo mount --bind /$d /mnt/$d; done

# 5. enter it
sudo chroot /mnt /bin/bash

sudo systemd-nspawn -D /mnt              # a container-like chroot, binds handled

grub2-mkconfig -o /boot/grub2/grub.cfg          # regenerate the menu
grub2-install /dev/sda                          # BIOS: write to the disk's MBR
dnf reinstall grub2-efi shim                    # UEFI: replace the EFI binaries
dracut -f --regenerate-all                      # rebuild every initramfs
passwd root                                     # reset a lost password
vi /etc/fstab                                   # fix the entry that stranded it
touch /.autorelabel                             # if SELinux is enforcing

efibootmgr -v                             # what the firmware will try, in order
efibootmgr -o 0003,0001                   # change the order

Across distributions

The stages are the same and the tooling around them is not, which matters because a boot repair is the one job where you cannot look anything up on the broken machine.

The initramfs row is the one to memorise. Changing the storage under root, adding LVM or LUKS, or swapping a disk controller means the initramfs no longer contains a driver it needs, and the machine panics on mount with a message that names the root device rather than the missing module. dracut -f on one family and update-initramfs -u on the other is the fix, and running the wrong one gives you a command not found on a machine you are already fighting.

The /.autorelabel row catches people restoring a RHEL system from a rescue environment. Files written or moved outside the running policy get whatever label the rescue environment gave them, so the machine boots and services then fail on denials that look nothing like the original fault.

Prove it

From a machine that still boots, so the answers exist before you need them:

# Where is root, and what is the kernel actually being told
cat /proc/cmdline
findmnt /

# Do fstab and reality agree, before the next reboot proves they do not
findmnt --verify
sudo mount -a

# Which target will it boot into
systemctl get-default

# Did anything fail on the way up
systemctl --failed
systemd-analyze
systemd-analyze blame | head

From a rescue shell, the order is narrower:

lsblk -f                      # what disks exist and what is on them
blkid                         # the UUIDs fstab and the kernel line refer to
journalctl -b -1 -p err       # why the last boot failed, if persistent

findmnt --verify and mount -a are the two seconds that prevent the whole category. An fstab typo costs nothing while you still have a shell, and costs an evening once the machine is sitting in emergency mode asking for a root password nobody wrote down.

What trips people up

1. Editing grub.cfg directly

It is generated, so the next kernel update overwrites it and the change vanishes at the least convenient moment. Edit /etc/default/grub, or drop a file into /etc/grub.d, then regenerate with the command your family uses.

2. Assuming a menu edit persists

Pressing e at the GRUB menu changes one boot and is never written to disk. That is a feature: it is how you test a parameter safely. It is also why the fix disappears on the next restart if you forget to make it permanent afterwards.

3. Confusing rescue with emergency

rescue.target is single user with filesystems mounted and most of the system initialised. emergency.target mounts root read-only and starts almost nothing. Reaching for emergency when rescue would do means doing the mounting by hand for no reason.

4. Being locked out by the root password prompt

Both rescue and emergency ask for it, and a cloud image frequently has no root password set at all. init=/bin/bash on the kernel line skips systemd entirely and hands you a shell, which is the route in when the prompt cannot be satisfied. Remember that root is mounted read-only at that point.

5. A chroot that misbehaves

Tools inside a chroot need the kernel interfaces the host is already providing. Bind mount /dev, /proc, /sys, and /run into the target before entering, or grub2-mkconfig produces a configuration built from the rescue environment’s view of the world rather than the target’s.

6. Reading degraded as a failure to boot

It means the target was reached and something under it failed. The machine is up and one unit is not, so systemctl --failed names it in one command. It is a report, not a boot problem.

Work it through

A server was rebooted for the first time in eight months after a routine update. It drops to an emergency shell with:

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot.
Cannot open access to console, the root account is locked.

Reason it out before reading on.

The second line is the one that matters. Emergency mode is the symptom. The locked root account is the immediate obstacle, and it means the prompt in front of you cannot be satisfied at all. Reboot, press e at the GRUB menu, and append init=/bin/bash to the kernel line to get a shell without systemd asking anybody for a password.

Remount root before trying to fix anything. The shell you land in has root mounted read-only, so every edit fails in a way that looks like a permission problem and is not:

mount -o remount,rw /

Eight months of uptime is itself a clue. Eight months of uptime means the cause was introduced long ago and only takes effect on a restart, which points hard at fstab:

findmnt --verify
cat /etc/fstab
lsblk -f            # do the UUIDs in fstab still exist

A device that was renamed, removed, or reformatted since the last boot leaves an fstab line pointing at a UUID that no longer exists. local-fs.target fails, and emergency mode is systemd doing exactly what it should.

Fix it in a way that cannot recur. Correct the UUID, and add nofail to anything that is not required for the machine to function, so a missing volume degrades one mount rather than the whole boot.

The general lesson is about timing rather than about fstab. A change made in January and a failure in September are the same event, because the boot path is only exercised at boot. Anything you edit there is untested until a restart, which is the argument for running mount -a and findmnt --verify at the moment you make the change rather than months later.

Try it

Only on a VM with a snapshot, and take the snapshot first. This is the one topic in the track where the exercise can genuinely leave a machine unbootable, which is also why it is worth doing.

1. Snapshot. Then add a line to /etc/fstab mounting a UUID that does not exist, without nofail. Reboot and watch where you land.
2. Recover it. Use the emergency shell if you have a root password, and init=/bin/bash if you do not. Remember the read-only remount.
3. Restore the snapshot, add the same broken line with nofail, and reboot again. Note that the machine now comes up with one mount missing rather than no machine at all.
4. Boot once with systemd.unit=rescue.target and once with systemd.unit=emergency.target. Run findmnt in each and compare what is mounted.

Verification step. You have step 4 right when you can say, without checking, which of the two would let you edit a file on a separate /var partition straight away and which would need you to mount it first.

For the exam

Where it stops tells you the stage. No GRUB menu means firmware, disk, or bootloader. A menu means the kernel can run and you can edit its parameters.

e at the GRUB menu edits an entry for one boot only.

root=UUID= on the kernel command line says where the root filesystem is, and a stale UUID is a common cause of a kernel panic on mount.

systemd.unit=rescue.target is single user with filesystems mounted; emergency.target starts almost nothing and mounts root read-only.

Both ask for the root password. init=/bin/bash does not, which is the route in when no root password exists.

rd.break gives a shell in the initramfs, with the real root at /sysroot.

Rebuild an initramfs with dracut -f after changing storage hardware or adding LVM or LUKS under root.

degraded means the target was reached with failures. systemctl --failed names them.

systemd-analyze splits boot into kernel, initrd, and userspace. A 90 second unit is a timeout, not work.

A chroot needs /dev, /proc, /sys, and /run bind mounted or the tools inside it will misbehave.

On UEFI, the bootloader lives on the EFI system partition and efibootmgr manages the firmware’s entries.

Where this sits

Lesson 09 walked the boot sequence when it works. This lesson is the same sequence when it does not, and the stage that failed decides which tool applies. Lesson 67 owns the filesystem and fstab problems that produce most emergency shells, and lesson 69 takes over once you have a prompt and a failed unit.

That completes block F, and with it the material for all five domains.

References

• systemd.special(7), rescue and emergency targets - freedesktop.org. Accessed 2026-08-09.
• systemd-analyze(1) - freedesktop.org. Accessed 2026-08-09.
• GNU GRUB manual - GNU. Accessed 2026-08-09.
• dracut(8) - man7.org. Accessed 2026-08-09.
• The kernel’s command-line parameters - kernel.org. Accessed 2026-08-09.

The commands here were run on a real machine, not written from memory. The transcripts come from Fedora CoreOS 44.20260707.3.1 on aarch64. The kernel command line is long because that is an ostree system, and it is shown as it actually is rather than trimmed to look tidy. The degraded state is genuine: that machine really did have rpm-ostreed.service failed at the time of capture, which is exactly the situation the section describes, and it had been giving a normal login prompt throughout.

Pictures. The screenshots on this page are freely licensed files from Wikimedia Commons, downloaded and served from this site rather than linked across to somebody else’s server. Both are unaltered.

• Grub edit boot menu by Svkeulen, CC BY-SA 3.0.
• Linux 5.7 kernel panic by Adhiansyah Ancha, GPL v2 or later.