Hardware and kernel issues

Before you read. The kernel log has three lines in it about dm-0, a device nobody configured, and the application is throwing errors that mention nothing of the sort.

Somewhere below the software there is a piece of hardware, or something pretending to be one, and it has stopped cooperating.

Hardware problems are the ones people reach for last, which is usually correct because they are the least common. When they do occur they are unmistakable if you know the vocabulary, and baffling if you do not, because the kernel names devices and layers rather than the applications you recognise.

This lesson is about reading those messages, and about the harder judgement underneath: deciding whether the hardware is genuinely at fault before somebody spends a day replacing a healthy disk.

Some words you will need

ring buffer

The kernel's in-memory log. What dmesg prints.

block layer

The kernel between filesystems and storage drivers. Reports as blk_update_request.

SMART

Self-monitoring built into drives. The disk's own view of its health.

taint

A flag recording that something happened which makes the kernel's behaviour unsupportable.

MCE

Machine check exception. The CPU reporting a hardware error to the operating system.

ECC

Memory that detects and corrects single-bit errors, and reports them.

device-mapper

The kernel layer behind LVM, encryption, and RAID. Its devices appear as dm-N.

What breaks without this

Healthy hardware gets replaced. A driver problem or a cable produces identical symptoms to a failing disk, and the disk is what gets swapped.

Failing hardware gets ignored. Correctable errors are logged for weeks before the uncorrectable one, and nobody was reading the log.

Data is written to a dying disk. Every repair attempt and every retry makes recovery less likely.

Support declines the case. A tainted kernel means the vendor cannot reproduce your configuration, and that is the first thing they check.

The wrong layer is blamed. An application error about a file is the last link in a chain that started at the controller.

What a real I/O error looks like

Rather than describing one, here is a device built to fail. Device-mapper has an error target that returns a failure for every read, which produces the genuine article without harming anything.

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ sudo dmsetup remove faultydisk 2>/dev/null
echo "--- a block device whose every read fails, built with device-mapper ---"
sudo dmsetup create faultydisk --table "0 2048 error"
sudo dd if=/dev/mapper/faultydisk of=/dev/null bs=512 count=1 2>&1 | tail -3
echo "--- what the kernel logged about it ---"
sudo dmesg | tail -3
sudo dmsetup remove faultydisk
--- a block device whose every read fails, built with device-mapper ---
0+0 records in
0+0 records out
0 bytes copied, 8.8084e-05 s, 0.0 kB/s
--- what the kernel logged about it ---
[30718.870606] Buffer I/O error on dev dm-0, logical block 0, async page read
[30718.870633] Buffer I/O error on dev dm-0, logical block 0, async page read
[30718.882536] Buffer I/O error on dev dm-0, logical block 0, async page read

Look at what dd said: 0 bytes copied, no error text of its own worth reading. The tool that failed is nearly silent. The kernel is where the explanation is.

Buffer I/O error on dev dm-0, logical block 0 names three things: the layer that noticed, the device, and the exact block. That precision is the point of reading dmesg rather than application logs.

Learn the shape of these messages, because the prefix tells you the layer:

The distinction that matters most is between a device that returned an error and a device that stopped answering. An error means the hardware is working well enough to say no. A timeout means it stopped responding entirely, which is more often a controller, a cable, power, or firmware than the media itself.

And dm-0 deserves a note, because it is the name people find least helpful. Device-mapper numbers its devices, so the message tells you nothing about which volume it is. Translate it:

sudo dmsetup ls                    # names against dm-N
lsblk                              # the whole tree, names included
ls -l /dev/mapper/                 # symlinks pointing at dm-N

Present but not working

The other family of hardware faults is the device that exists and does nothing, and the diagnostic question is whether the kernel found it, bound a driver to it, and brought it up. Those are three separate steps.

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- what the kernel found on the PCI bus ---"; lspci 2>/dev/null | head -5 || echo "(no lspci on this image)"; echo "--- memory the firmware handed over ---"; sudo dmesg | grep -iE "Memory:|efi:" | head -3
--- what the kernel found on the PCI bus ---
00:00.0 Host bridge: Apple Inc. Device 1a05
00:01.0 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01)
00:05.0 Communication controller: Red Hat, Inc. Virtio 1.0 console (rev 01)
00:06.0 Mass storage controller: Red Hat, Inc. Virtio 1.0 block device (rev 01)
00:07.0 Mass storage controller: Red Hat, Inc. Virtio 1.0 file system (rev 01)
--- memory the firmware handed over ---

Five devices, and the mixture is worth noticing: an Apple host bridge because this is a virtual machine on Apple hardware, and Red Hat virtio devices for everything the hypervisor provides. On a physical server the same command lists real controllers and network cards.

lspci listing a device proves only that the bus enumerated it. The next question is whether a driver claimed it, and -k answers that:

lspci -k                    # each device with its kernel driver, if any
lspci -nn                   # with numeric vendor:device IDs, for searching

A device with no Kernel driver in use line is the classic “present but not working” case. It means the hardware is fine and the software to operate it is missing, which is a firmware package, a missing module, or a kernel too old for the part.

The sequence to work through:

Missing firmware is worth calling out because the message is explicit and people still miss it. Many network and graphics devices need a binary blob loaded at initialisation, and dmesg will say Direct firmware load for ... failed with error -2. The device then exists and does nothing. The fix is a package, usually linux-firmware, and no amount of hardware replacement helps.

sudo smartctl -H /dev/sda           # overall pass or fail
sudo smartctl -a /dev/sda           # every attribute
sudo smartctl -l error /dev/sda     # the drive's own error log
sudo smartctl -t short /dev/sda     # start a self test, minutes
sudo smartctl -l selftest /dev/sda  # results

sudo smartctl -a /dev/nvme0        # or: sudo nvme smart-log /dev/nvme0

sudo mount -o remount,ro /mnt/data

sudo ddrescue -f -n /dev/sdb /dev/sdc /var/tmp/rescue.map     # fast pass, skip errors
sudo ddrescue -d -f -r3 /dev/sdb /dev/sdc /var/tmp/rescue.map # retry the gaps

cat /proc/mdstat                # software RAID
sudo mdadm --detail /dev/md0    # per-device state

Taint

The kernel keeps a flag recording whether anything has happened that makes its behaviour hard to reason about.

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- is the kernel tainted, and by what ---"; cat /proc/sys/kernel/tainted; echo "--- CPU, as the kernel sees it ---"; lscpu | grep -E "^(Architecture|CPU\(s\)|Model name|Vendor ID|Hypervisor)" 2>/dev/null
--- is the kernel tainted, and by what ---
0
--- CPU, as the kernel sees it ---
Architecture:                            aarch64
CPU(s):                                  5
Vendor ID:                               Apple
Model name:                              -

Zero, which is what a clean kernel reports. The value is a bitmask, so anything non-zero needs decoding:

cat /proc/sys/kernel/tainted                                    # the number
sudo dmesg | grep -i taint                                      # why, in words

The bits worth recognising:

Bits 4, 5, and 7 are the ones that matter for this lesson, because they are the kernel recording that something genuinely went wrong rather than that you installed a driver.

Why anybody cares: taint is the first thing a vendor’s support process checks. A kernel running a proprietary or out-of-tree module is not the kernel they ship, so they cannot reproduce your problem and will usually ask you to remove it before continuing. That is a reasonable position and it is worth knowing before you open the case rather than after.

sudo edac-util -v                          # per-DIMM correctable and uncorrectable
sudo ras-mc-ctl --summary                  # if rasdaemon is running
sudo dmesg | grep -iE 'EDAC|Machine check|mce|Hardware Error'
sudo mcelog --client                       # on older systems

Across distributions

The kernel reports faults identically everywhere, because it is the same kernel. What varies is whether the tools to read those reports are installed and whether the firmware the hardware needs shipped with the system at all.

The firmware row is the one that produces a mystery. Debian split non-free firmware out of the default install for years on licensing grounds, so a network card or a GPU that works immediately on a RHEL machine comes up dead on a Debian one with Direct firmware load for ... failed with error -2 in dmesg. The hardware is fine, the driver is loaded, and the blob it wants to push into the device is simply not on the disk. Recent Debian installers offer to include it, which means you will meet both behaviours depending on how the machine was built.

smartmontools being absent by default on both families is worth knowing before you need it, because the moment you want SMART attributes is usually the moment the machine is misbehaving and you would rather not be installing packages onto it.

Prove it

Hardware faults appear in the kernel log and essentially nowhere else, so this is a short list:

# The kernel's own account, most recent last
sudo dmesg -T | tail -50
sudo journalctl -k -p err -b

# Did the kernel see the device, and did a driver bind to it
lspci -k | grep -A3 -i <device>
lsusb

# Is a firmware blob missing
sudo dmesg | grep -i firmware

# Storage health, on real disks only
sudo smartctl -H /dev/sda
sudo smartctl -A /dev/sda | grep -iE "reallocated|pending|uncorrect|crc"

# Has anything compromised the kernel's supportability
cat /proc/sys/kernel/tainted

A clean dmesg is a real result. Hardware that is genuinely failing is loud: resets, timeouts, medium errors, correctable ECC counts climbing. A machine behaving badly with nothing in the kernel log is evidence that the fault is above the hardware, and ruling out a whole layer in one command is worth doing early rather than after an afternoon of guessing.

What trips people up

1. Reading a PASSED health check as a healthy drive

The overall health line is one threshold test the firmware performs, and it stays at PASSED until an attribute crosses the vendor’s own limit. Reallocated_Sector_Ct and Current_Pending_Sector can be climbing steadily, with real read failures reaching the kernel, while that summary still says PASSED. Read the attributes and their trend.

2. Expecting SMART data from a virtual disk

A cloud volume or a hypervisor-provided disk has no SMART to report, so smartctl returns nothing useful and that says nothing about the underlying hardware, which belongs to somebody else. The equivalent evidence is I/O errors in dmesg and whatever the provider’s own status page offers.

3. Treating an error and a timeout as the same fault

A device that returns an error is answering: it received the request, tried, and failed, which points at the media. A device that times out has stopped responding, which points at the controller, the cable, or power. The second is more often something other than the disk itself.

4. Blaming the drive when the counter says cable

UDMA_CRC_Error_Count records corruption on the link between the controller and the drive, not on the platters. It climbs with a bad cable, a loose connector, or a backplane problem, and replacing a perfectly good disk leaves it climbing on the new one.

5. Assuming a listed device has a working driver

lspci shows what the bus enumerated, which is a question about the hardware being present. lspci -k shows whether a kernel module claimed it. A device listed with no Kernel driver in use line is visible and unusable, and the cause is a missing module or missing firmware rather than a fault.

6. Ignoring correctable ECC errors

By definition they were corrected, so nothing broke and it is tempting to move on. A rising rate of correctable errors on one DIMM is the standard warning that it is going to produce an uncorrectable one, which is the sort that takes the machine down without a log entry explaining why.

Work it through

A database server has been slow for two days and the application team reports occasional query timeouts. Nothing changed. top shows low CPU and the load average is high.

Reason it out before reading on.

Ask the kernel before asking anything else. High load with idle CPU means processes blocked rather than computing, and blocked usually means storage:

sudo dmesg -T | tail -50

Say it contains repeated lines like blk_update_request: I/O error, dev sda, sector 1234567 op 0x0:(READ) alongside ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen.

The device is answering badly rather than not answering. The device is returning errors and the link is being reset, which is the hardware answering badly rather than not answering. Each reset stalls every request in flight, which is exactly what produces intermittent timeouts on an otherwise idle machine.

Get the drive’s own account of itself:

sudo smartctl -H /dev/sda
sudo smartctl -A /dev/sda | grep -iE "reallocated|pending|crc"

Two readings, two different conclusions. Current_Pending_Sector climbing means the media is failing and the drive should be replaced. UDMA_CRC_Error_Count climbing with the other attributes flat means the link is at fault, so reseat or replace the cable before condemning a healthy disk.

PASSED is not where to stop. If the health line says PASSED and the attributes are climbing, the attributes win. The threshold has not been crossed yet, and the trend is what tells you when it will be.

The general lesson is about which layer to interrogate first. The report was about query timeouts, and nothing about the diagnosis went near the database. The load average being high while the CPU was idle pointed one layer down, and one dmesg decided it.

Try it

Optional, and most of it is read-only, so it is safe on a machine you care about.

1. Run sudo dmesg -T | less on a machine that has been up for a while and read it from the top. The boot-time hardware enumeration is the best free tour of what the kernel thinks the machine is.
2. Run lspci -k and find a device with no Kernel driver in use line. On a VM there is usually at least one. Work out whether that matters.
3. Run sudo smartctl -A on a real disk and write down Reallocated_Sector_Ct, Current_Pending_Sector, and UDMA_CRC_Error_Count. That is your baseline, and the numbers mean nothing without one. On a VM, confirm what the command says instead and why.
4. Read /proc/sys/kernel/tainted. If it is not zero, decode it against the kernel documentation and work out which module did it.

Verification step. Step 3 is complete when you can say what those three numbers were and, more usefully, where you wrote them down. A SMART attribute read once during an incident is nearly useless, because the question is always whether it is rising.

For the exam

dmesg and journalctl -k are where hardware errors appear, not in application logs.

blk_update_request: I/O error names the device and sector. Media problems are critical medium error.

A device that returns an error is answering; a device that times out has stopped. The second is more often the controller, cable, or power.

lspci shows enumeration, lspci -k shows whether a driver bound. No driver means missing module or firmware.

Direct firmware load ... failed means a missing firmware package, usually linux-firmware.

SMART: Reallocated_Sector_Ct and Current_Pending_Sector predict failure. UDMA_CRC_Error_Count means the cable.

A PASSED health check does not mean a healthy drive. Read the attributes.

Virtual and network-attached disks have no SMART data.

A tainted kernel is recorded in /proc/sys/kernel/tainted, and vendors check it before accepting a support case.

ECC corrects and logs single-bit errors. Correctable errors rising is the warning to act on.

A completely clean dmesg is evidence against a hardware fault.

Where this sits

Lesson 10 covered the kernel and its modules, and lesson 11 covered device discovery. This lesson is the same territory when something has gone wrong. Lesson 67 handles the filesystem sitting on top of a failing device, and lesson 76 covers a disk that is slow rather than broken, which is a different investigation with a different answer.

References

• dmesg(1) - man7.org. Accessed 2026-08-09.
• smartctl(8) - smartmontools. Accessed 2026-08-09.
• Kernel documentation: tainted kernels - kernel.org. Accessed 2026-08-09.
• Kernel documentation: machine check exceptions - kernel.org. Accessed 2026-08-09.
• dmsetup(8) - man7.org. Accessed 2026-08-09.

The commands here were run on a real machine, not written from memory. The transcripts come from Fedora CoreOS 44.20260707.3.1 on aarch64. The I/O errors are genuine kernel messages, produced by building a device-mapper error target so that every read really did fail, and the device was removed in the same command that created it. The lspci listing shows an Apple host bridge beside Red Hat virtio devices because that is what this virtual machine actually is. smartctl is not installed on that image, which is the reason the SMART section carries no transcript and says so.