Disk space and inode problems

Before you read. The alert says the disk is full. You log in, run df, and it agrees: 80% used, climbing. You run du to find the culprit and it reports that the filesystem contains almost nothing.

Both commands are working correctly. They are answering different questions, and the gap between the two answers is the diagnosis.

Disk-space problems are common, and they are one of the few troubleshooting areas where the tools will hand you the answer in about thirty seconds if you know which two to run and in what order. The reason people find them frustrating is that the obvious command is often the wrong one.

Some words you will need

inode

The structure holding a file's metadata and the pointers to its data. Every file uses exactly one.

link count

How many directory entries point at an inode. When it reaches zero and nothing has it open, the space is freed.

file descriptor

A process's open handle to a file. Keeps the inode alive regardless of directory entries.

unlink

What rm actually does: remove the name, not the data.

reserved blocks

A percentage of the filesystem only the superuser may write into.

sparse file

A file with holes: it reports a large size but only occupies the blocks actually written.

ENOSPC

The error behind "No space left on device". Means out of blocks or out of inodes.

What breaks without this

Writes fail everywhere at once. A full filesystem is not a gradual degradation. Databases stop committing, logs stop being written, and package operations abort partway.

Deleting the obvious file changes nothing. The space stays gone, so the next instinct is to reboot, which does work and teaches you nothing.

The machine boots into emergency mode. A full / can prevent services starting at all, which turns a disk problem into an outage.

Nobody can log in. Session setup needs to write, so a full / or /home can lock out the people who would fix it.

And the same alert fires again next week, because the file that filled the disk is still being written to by something nobody identified.

Two commands, two questions

Start here, because everything else in this lesson follows from it.

df asks the filesystem how many blocks are allocated. It reads the filesystem’s own accounting, so it counts every block in use whether or not a name points at it.

du walks the directory tree and adds up the files it finds. It counts what it can see by name.

Most of the time those produce the same number:

# AlmaLinux 10.2, aarch64
$ mkfs.ext4 -q $DEV0; mkdir -p /mnt/d; mount $DEV0 /mnt/d; dd if=/dev/zero of=/mnt/d/big.log bs=1M count=40 status=none; echo "--- df and du agree ---"; df -h /mnt/d | tail -1; du -sh /mnt/d
--- df and du agree ---
/dev/loop0       55M   41M   11M  80% /mnt/d
41M	/mnt/d

41M by both counts. Nothing to explain.

The space that is not in any file

Now the same filesystem, except a process opens the log file before it is deleted. This is not a contrived situation: it is precisely what happens when somebody clears a log that a running daemon still has open.

# AlmaLinux 10.2, aarch64
$ mkfs.ext4 -q $DEV0; mkdir -p /mnt/d; mount $DEV0 /mnt/d; dd if=/dev/zero of=/mnt/d/big.log bs=1M count=40 status=none; sleep 300 < /mnt/d/big.log & sleep 1; rm /mnt/d/big.log; echo "--- du now sees an empty filesystem ---"; du -sh /mnt/d; echo "--- but df still says the space is gone ---"; df -h /mnt/d | tail -1
--- du now sees an empty filesystem ---
13K	/mnt/d
--- but df still says the space is gone ---
/dev/loop0       55M   41M   11M  80% /mnt/d

13K against 41M. That is the entire phenomenon, and both numbers are correct.

rm does not delete a file. The system call is called unlink, and the name is accurate: it removes a directory entry. The kernel frees the data only when both the link count is zero and no process has the file open.

Here the link count is zero, so du cannot find the file: there is no name to walk to. But a process still holds a descriptor, so the inode and all 40 MB of its blocks remain allocated, which is what df reports.

The rule that follows: when df and du disagree, df is right about the space and du is right about the files. The difference is space held by something with no name.

Finding it

lsof lists open files, and +L1 restricts that to files with a link count below 1, which is exactly the set you want:

# AlmaLinux 10.2, aarch64
$ mkfs.ext4 -q $DEV0; mkdir -p /mnt/d; mount $DEV0 /mnt/d; dd if=/dev/zero of=/mnt/d/big.log bs=1M count=40 status=none; sleep 300 < /mnt/d/big.log & sleep 1; rm /mnt/d/big.log; echo "--- lsof names the process still holding it ---"; lsof +L1 /mnt/d 2>/dev/null
--- lsof names the process still holding it ---
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
sleep    41 root    0r   REG    7,0 41943040     0   12 /mnt/d/big.log (deleted)

Read the columns, because each one is part of the answer:

NLINK 0 and (deleted) are the confirmation. If lsof +L1 returns nothing, this is not your problem and you should be looking elsewhere.

Getting the space back

The obvious fix is to restart the process holding the descriptor, and usually that is the right one: the space is released the moment the last descriptor closes.

When restarting is expensive, there is another way. The descriptor is reachable through /proc/<pid>/fd/, and truncating it frees the blocks while the process keeps running:

# AlmaLinux 10.2, aarch64
$ mkfs.ext4 -q $DEV0; mkdir -p /mnt/d; mount $DEV0 /mnt/d; dd if=/dev/zero of=/mnt/d/big.log bs=1M count=40 status=none; sleep 300 < /mnt/d/big.log & P=$!; sleep 1; rm /mnt/d/big.log; echo "--- space still held ---"; df -h /mnt/d | tail -1; echo "--- truncate the descriptor without killing the process ---"; : > /proc/$P/fd/0; df -h /mnt/d | tail -1
--- space still held ---
/dev/loop0       55M   41M   11M  80% /mnt/d
--- truncate the descriptor without killing the process ---
/dev/loop0       55M   14K   51M   1% /mnt/d

41M to 14K, and the process never stopped. 80% to 1% in one command.

Understand what this does before you reach for it. The data is discarded immediately. If the process is a database writing its journal, you have just destroyed the journal. It is the correct move for a log file that has already been rotated and is being written to by mistake, and the wrong move for anything you might have wanted to read.

mount --bind / /mnt/root-real
du -sh /mnt/root-real/var/log
umount /mnt/root-real

Running out of inodes instead of space

The second way a filesystem refuses to write has nothing to do with space at all.

Every file consumes one inode, and on ext4 the number of inodes is fixed when the filesystem is created. Run out of them and the filesystem is full in the only sense that matters, whatever df says about blocks.

# AlmaLinux 10.2, aarch64
$ mkfs.ext4 -q -N 256 $DEV0; mkdir -p /mnt/d; mount $DEV0 /mnt/d; i=0; while [ $i -lt 400 ]; do if ! (: > /mnt/d/f$i) 2>/dev/null; then break; fi; i=$((i+1)); done; echo "created $i files"; echo "--- space? plenty ---"; df -h /mnt/d | tail -1; echo "--- inodes? none ---"; df -i /mnt/d | tail -1
created 245 files
--- space? plenty ---
/dev/loop0      6.9M   18K  6.3M   1% /mnt/d
--- inodes? none ---
/dev/loop0        256   256     0  100% /mnt/d

1% full by space. 100% full by inodes. df -h alone would tell you everything is fine.

And here is the error, which is the part that misleads people:

# AlmaLinux 10.2, aarch64
$ mkfs.ext4 -q -N 256 $DEV0; mkdir -p /mnt/d; mount $DEV0 /mnt/d; i=0; while [ $i -lt 400 ]; do if ! (: > /mnt/d/f$i) 2>/dev/null; then break; fi; i=$((i+1)); done; echo "--- and this is the error you get, on a filesystem that is 1% full ---"; touch /mnt/d/one-more
--- and this is the error you get, on a filesystem that is 1% full ---
touch: cannot touch '/mnt/d/one-more': No space left on device

“No space left on device” on a filesystem with 6.3 MB free. The kernel returns ENOSPC for both conditions and the message only mentions one of them.

So df -i belongs in your reflexes right next to df -h. Any time you see ENOSPC, run both. It costs nothing and it eliminates half the possibilities.

What exhausts inodes in real life: a mail queue, a session directory, a cache with one file per key, a build system that never cleans up, PHP sessions, and anything that writes one small file per event. The signature is a directory containing hundreds of thousands of tiny files.

Finding the offender means counting rather than measuring:

# where are the files, by count rather than by size
sudo find /var -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20

-xdev keeps find on one filesystem, which matters: without it you walk into /proc, /sys, and every other mount, and the answer is nonsense.

mkfs.ext4 -i 4096 /dev/sdb1     # one inode per 4 KB, four times the default

sudo tune2fs -l /dev/sdb1 | grep -i 'reserved block count'
sudo tune2fs -m 1 /dev/sdb1      # reduce to 1 percent

Finding what is actually using the space

When df and du agree and the filesystem really is full, the job is ordinary: find the big things.

-x and -xdev are not optional. Both mean “stay on this filesystem”. A du that wanders into /proc, /sys, and every network mount produces numbers that mean nothing and takes minutes doing it.

Descend, do not guess. du --max-depth=1 at each level, following the largest number down, finds a directory in four or five steps without reading thousands of lines.

The usual suspects, in rough order of likelihood: /var/log, a journal that was never capped, /var/lib/docker or /var/lib/containers, a package manager cache, a core dump in /var/lib/systemd/coredump, a developer’s build output, and a database’s write-ahead log that stopped being checkpointed.

The order to work in

When the disk is full, this sequence gets you to the cause quickly and does not depend on a guess:

1. df -h: which filesystem, and how full.
2. df -i: blocks or inodes? These two together eliminate half the possibilities in two seconds.
3. du -xh --max-depth=1 <mountpoint>: descend toward the largest number.
4. If du cannot account for it, lsof +L1 <mountpoint>: deleted-but-open files, and the process to restart.
5. If neither explains it, check for files under a mount point by bind-mounting the parent.
6. Fix the cause, not the symptom. Something wrote all that. Rotation, retention, or a cap on the journal is the actual repair.

And write down what you found, per lesson 63, because a disk that filled once will fill again and the next person deserves the four-line version rather than the investigation.

Across distributions

The commands are the same everywhere. What differs is the filesystem you are standing on, and that decides whether one of these failures can happen to you at all.

The inode row is the reason this topic splits by family. ext4 decides how many inodes the filesystem will ever have when it is created, based on a bytes-per-inode ratio, and nothing can raise it afterwards. A mail spool or a cache directory full of tiny files therefore exhausts the inode table long before it exhausts the blocks, and the error is No space left on device on a filesystem df -h reports as 40 percent used. XFS allocates inodes as it needs them, so on a RHEL root you will very likely never meet this.

The 5 percent reservation catches people in the other direction. An ext4 filesystem reports itself full to an unprivileged writer while df still shows five percent free, because that slice is held back for root and for reducing fragmentation. On a large data volume with no root processes writing to it, that reservation is a lot of disk doing nothing, and tune2fs -m 1 reclaims most of it.

Prove it

Two commands answer two different questions, and running both is what makes the disagreement between them informative:

# The filesystem's own accounting, and the inode table separately
df -h
df -i

# What the directory tree actually contains, staying on one filesystem
sudo du -xh --max-depth=1 / | sort -rh | head

# The gap between them: files with no name left, held open by a process
sudo lsof +L1

# Who is holding the largest of them
sudo lsof -nP | awk '$5=="REG" && $7>1073741824'

# On ext4, how much is reserved rather than missing
sudo tune2fs -l /dev/sdb1 | grep -i "reserved block"

The -x in du is not optional. Without it, du walks straight through mount points into other filesystems, so the totals include volumes that are not the one that is full, and the number you get back cannot be compared against df. The same applies to find, where the flag is spelled -xdev.

What trips people up

1. Believing du when it disagrees with df

They measure different things. df asks the filesystem how many blocks are allocated; du adds up the files it can find by name. A file that has been deleted while a process still holds it open has no name, so du cannot see it and its blocks are still allocated. That gap is the answer, not an error.

2. rm on a log file that is open

Removing a name from a directory does not free anything while a process holds a descriptor to the inode. The log keeps growing, invisibly, and the space comes back only when the service is restarted. Truncating in place with > file or truncate -s 0 file frees the space immediately and leaves the descriptor valid, which is why logrotate uses copytruncate when it cannot signal the daemon.

3. Reading No space left on device as a block problem

It is the same errno for blocks and inodes. df -h looking healthy does not clear the error, it just tells you which of the two to look at. Run df -i before theorising.

4. Deleting files to fix a full disk without finding the cause

The disk fills again, usually faster, because whatever was writing is still writing. Find the growth first, then reclaim. A filesystem that went from 40 to 100 percent in a day has a cause with a name.

5. Forgetting that a mount can hide a full filesystem

Writing into a directory whose intended volume failed to mount fills the parent filesystem instead, silently and at full speed. findmnt on the path is the check, and it is a common way for a root filesystem to fill overnight.

6. Running du from / without -x

On a machine with network mounts, that walks the NFS server as well, which is slow, misleading, and occasionally noticed by whoever owns the NFS server.

Work it through

A monitoring alert says / is at 98 percent on a web server. du -sh /* adds up to about half the reported usage. The service is still running.

Reason it out before reading on.

Which resource is short decides everything that follows. Two seconds, and it decides everything that follows:

df -h /
df -i /

Say blocks are at 98 percent and inodes at 12 percent. It is a space problem rather than a file-count problem.

Half the space unaccounted for is not a rounding error. Half the space unaccounted for is not a rounding error, and there are only a few explanations. The most common by a distance is a deleted file still held open:

sudo lsof +L1

Say that returns /var/log/app/debug.log (deleted) at 22 GB, held by the application. Somebody removed the log to free space, the process kept its descriptor, and the file has been growing invisibly ever since.

Reclaim it without losing the service. Restarting releases the descriptor and frees the space, and there is a way to do it without a restart at all if the process must stay up:

# find the descriptor number, then truncate through /proc
sudo ls -l /proc/<pid>/fd | grep deleted
sudo truncate -s 0 /proc/<pid>/fd/3

The log being deleted by hand is the actual defect. The log was being deleted by hand because it grows without bound, which means log rotation is either missing or not signalling the daemon. That is the actual defect, and until it is fixed this recurs every few weeks.

The reasoning worth keeping: the alert described a percentage, du and df disagreeing described the mechanism, and lsof +L1 named the file. None of those three steps involved guessing what was large.

Try it

Optional, and a container is enough. The first exercise is the one worth doing, because reading about deleted-but-open files never lands the way watching it does.

1. In one terminal, create a large file and hold it open: dd if=/dev/zero of=/tmp/big bs=1M count=500, then tail -f /tmp/big > /dev/null &. Note df -h /tmp.
2. Delete it with rm /tmp/big. Run df -h /tmp and du -sh /tmp and compare. The space is still gone and du can no longer see the file.
3. Find it with sudo lsof +L1, then kill the tail and check df again.
4. On an ext4 filesystem, build one with very few inodes on purpose: mkfs.ext4 -N 128 /tmp/tiny.img on a small file, mount it, and create empty files until it fails. Read the error text carefully.

Verification step. Step 4 is right when the error says No space left on device, df -h on that mount shows plenty of free space, and df -i shows the inode column at 100 percent. Being able to recognise that combination from the three commands is the whole point of the exercise.

For the exam

df reports the filesystem’s accounting. du walks the tree. When they disagree, suspect deleted-but-open files.

rm unlinks a name. Space is freed when the link count is zero and no process holds the file open.

lsof +L1 finds files with no remaining name.

“No space left on device” can mean inodes. Always run df -i as well as df -h.

ext4 fixes its inode count at mkfs time; XFS allocates dynamically.

ext4 reserves 5% for root by default, changed with tune2fs -m.

Use -x or -xdev to keep du and find on one filesystem.

Truncating in place (> file) is safe; rm on an open log is not.

Where this sits

Lesson 12 built filesystems and lesson 13 mounted them; this is what to do when one of them fills. Lesson 26’s find is the tool for locating the offender, and lesson 65’s journal is usually where the evidence of what wrote it lives.

The next lesson takes the same posture toward a service that claims to be running while the application behind it is not.

References

• df(1) - man7.org. Accessed 2026-08-09.
• du(1) - man7.org. Accessed 2026-08-09.
• lsof(8) - man7.org. Accessed 2026-08-09.
• unlink(2) - man7.org. Accessed 2026-08-09.
• tune2fs(8) - man7.org. Accessed 2026-08-09.

The commands here were run on a real machine, not written from memory. The transcripts come from AlmaLinux 10.2 on aarch64, with the filesystems built on loop devices so a 55 MB and an 8 MB filesystem could be filled quickly and thrown away. The deleted-but-open file really did hold 41 MB that du could not see, and truncating the descriptor really did return it without stopping the process. The inode filesystem was deliberately created with 256 inodes to reach exhaustion in a few hundred files rather than a few million.