RAID

Before you read. Hard disks fail. Not occasionally, predictably, at a rate you can put in a spreadsheet. A rack of forty disks will lose one or two a year and nobody considers that remarkable.

And yet servers stay up for years without anybody restoring a backup.

How does a machine keep serving files off a disk that has stopped working?

By not depending on any one disk. Several disks are presented to the operating system as one device, with enough duplication that losing a member is survivable. The filesystem above never notices.

The interesting part is the arithmetic underneath, how much capacity that costs, how many failures it survives, and what it does not protect you from, which is more than people assume.

Some words you will need

array

Several disks presented as one block device. On Linux, /dev/md0.

member

One of the disks in the array.

striping

Splitting data across disks so several can be read at once. Faster, no protection.

mirroring

Writing the same data to two disks. Protection, at half the capacity.

parity

Extra data from which any one missing piece can be recalculated. Protection, at the cost of one disk.

degraded

Running with a member missing. Still working, no longer protected.

What breaks without this

A single disk failure becomes an outage. Restoring from backup takes hours and loses everything since the last one.

Or you have RAID and do not know how to read it. An array can sit degraded for months, working perfectly, until the second disk goes and takes everything. The machine never complained because from its point of view nothing was wrong.

You trust it for the wrong thing. RAID protects against a disk failing. It does nothing about deletion, corruption, ransomware, or fire, and treating it as a backup is how organisations lose data while technically having redundancy.

The levels

RAID 0 is not RAID in the sense anyone means it. It has no redundancy and it doubles your risk: two disks, either one takes everything. The zero is appropriate.

RAID 5’s write cost is the thing the table cannot show. Changing one block means reading the old block and the old parity, computing the new parity, and writing both, four operations for one logical write. On a database that hurts. On a file archive nobody notices.

Parity is arithmetic, not a copy. Ap is computed from A1 and A2 such that any one of the three can be reconstructed from the other two. That is why n−1 disks of capacity protects n disks, and why a rebuild is CPU work rather than a straight copy.

Building an array

A RAID 1 mirror is created from two blank 512 MiB disks. Both are new and contain nothing.

# AlmaLinux 10.2, aarch64
$ mdadm --create /dev/md0 --level=1 --raid-devices=2 $DEVS --run; echo; cat /proc/mdstat
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Personalities : [raid1] 
md0 : active raid1 loop1[1] loop0[0]
      523264 blocks super 1.2 [2/2] [UU]
      [>....................]  resync =  0.1% (1024/523264) finish=8.3min speed=1024K/sec
      
unused devices: <none>

It resyncs anyway, and the array is usable the whole time. The kernel has no way to know the disks are blank, “empty” is not a thing a block device reports, so it copies every block from one to the other to guarantee they match. On two 512 MiB loop devices that takes minutes; on a pair of 20 TB disks it takes a day, and the array is degraded-but-working throughout.

--assume-clean skips it, and is safe only when you genuinely know both members are identical. Getting that wrong on a mirror means the two halves disagree and reads return whichever one they land on.

Learn to read /proc/mdstat, because it is the fastest health check there is and it is the same on every distribution.

• active raid1 loop1[1] loop0[0], the level and the members, with their index in the array.
• [2/2], two members expected, two present.
• [UU], one character per member. U is up, _ is missing.
• The progress bar is the initial resync, making both halves identical. The array is usable during it, just slower.

--run skips the confirmation prompt. The warning about metadata is worth reading once: version 1.2 metadata sits near the start of the member, which older bootloaders cannot see past, hence the note about /boot.

A three-disk RAID 5, and the arithmetic:

# AlmaLinux 10.2, aarch64
$ mdadm --create /dev/md0 --level=5 --raid-devices=3 $DEVS --run 2>/dev/null; sleep 3; mdadm --detail /dev/md0 | head -22
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
/dev/md0:
           Version : 1.2
     Creation Time : Sat Aug  8 01:12:45 2026
        Raid Level : raid5
        Array Size : 1044480 (1020.00 MiB 1069.55 MB)
     Used Dev Size : 522240 (510.00 MiB 534.77 MB)
      Raid Devices : 3
     Total Devices : 3
       Persistence : Superblock is persistent

       Update Time : Sat Aug  8 01:12:48 2026
             State : clean 
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

Three 512 MiB disks give a 1020 MiB array. Two disks’ worth of capacity, one disk’s worth spent on parity. That is the n−1 rule, in numbers you can check.

State: clean and Failed Devices: 0 are the two lines to read on a health check. mdadm --detail is the verbose view; /proc/mdstat is the one you glance at.

Losing a disk

# AlmaLinux 10.2, aarch64
$ cat /proc/mdstat | head -4; echo "--- pull a disk ---"; mdadm --manage /dev/md0 --fail $DEV0; cat /proc/mdstat | head -4; echo "--- is the data still there? ---"; cat /mnt/raid/important.txt
Personalities : [raid1] [raid4] [raid5] [raid6] 
md0 : active raid1 loop1[1] loop0[0]
      523264 blocks super 1.2 [2/2] [UU]
      
--- pull a disk ---
Personalities : [raid1] [raid4] [raid5] [raid6] 
md0 : active raid1 loop1[1] loop0[0](F)
      523264 blocks super 1.2 [2/1] [_U]
      
--- is the data still there? ---
payroll data

Putting it back

# AlmaLinux 10.2, aarch64
$ mdadm --manage /dev/md0 --remove $DEV0 >/dev/null; mdadm --manage /dev/md0 --add $DEV2; sleep 2; echo; cat /proc/mdstat | head -5; echo "--- wait for it to finish ---"; while grep -q recovery /proc/mdstat; do sleep 2; done; cat /proc/mdstat | head -4
mdadm: hot removed /dev/loop0 from /dev/md0
mdadm: added /dev/loop2

Personalities : [raid1] [raid4] [raid5] [raid6] 
md0 : active raid1 loop2[2] loop1[1]
      523264 blocks super 1.2 [2/1] [_U]
      [===============>.....]  recovery = 76.6% (401792/523264) finish=0.0min speed=200896K/sec
      
--- wait for it to finish ---
Personalities : [raid1] [raid4] [raid5] [raid6] 
md0 : active raid1 loop2[2] loop1[1]
      523264 blocks super 1.2 [2/2] [UU]
      

Two commands, then waiting. --remove then --add, and the rebuild starts on its own. Note it says recovery rather than resync, resync is making a new array consistent, recovery is rebuilding a replaced member. Same bar, different word, and the word tells you which situation you are in.

[UU] at the end is the acceptance test. Until the bar reaches 100% the array is still degraded and another failure is still fatal. On real hardware this is hours rather than seconds, and that window is the whole argument for RAID 6.

The whole replacement procedure:

--fail before --remove even on a dead disk. --remove refuses to take an active member, and a disk the kernel has not given up on yet still counts as active.

Hardware, software, and the thing in between

mdadm’s portability is underrated. The array metadata lives on the disks, so moving them to any Linux machine and running mdadm --assemble --scan brings the array up. A failed hardware controller, by contrast, frequently means sourcing the same model before you can read your own data.

Avoid firmware RAID. It is a BIOS option that looks like hardware RAID and is software RAID with a proprietary on-disk format and worse tooling. If the machine has it enabled, turning it off and using mdadm is almost always the better answer.

Across distributions

The config file path is the only real difference and it is enough to break a script that hardcodes one.

Prove it

Health, in one line, worth putting in a login banner or a monitoring check:

# The whole answer, if you can read it
cat /proc/mdstat

# The verbose version
sudo mdadm --detail /dev/md0

# Is monitoring actually going to tell anyone
systemctl status mdmonitor
sudo mdadm --monitor --scan --test    # sends a test alert now

Look for [UU], not for absence of errors. A degraded array produces no errors. The only difference between healthy and one-failure-from-catastrophe is an underscore.

What trips people up

1. The array says [_U] and nobody noticed

Degraded arrays work. Applications are fine, users are fine, performance is normal, and there is no redundancy left at all.

Monitor /proc/mdstat, and test that the monitoring works. An untested alert is not an alert.

2. --remove refuses

mdadm: hot remove failed for /dev/sdb: Device or resource busy. The member is still active as far as the kernel is concerned.

--fail it first, then --remove. Marking a dying disk as failed is safe and is what the command is for.

3. Treating RAID as a backup

It is not, and the distinction is worth stating precisely: RAID replicates writes. Delete a file and it is deleted on every member instantly. Encrypt the volume with ransomware and every copy is encrypted. Corrupt a database and the corruption is faithfully mirrored.

RAID protects against exactly one thing: a disk failing. Backups protect against everything else, and everything else is most of what happens.

4. The array does not come back after a reboot

Usually a missing or stale /etc/mdadm.conf. The array assembles under a different name, /dev/md127 is the classic, and /etc/fstab cannot find it.

mdadm --detail --scan >> /etc/mdadm.conf, then rebuild the initramfs. Use UUID= in fstab, which does not care what the array is called.

5. Rebuilding onto a smaller disk

Replacement disks must be at least as large as the smallest existing member. The same nominal size from a different manufacturer can be very slightly smaller, and mdadm will refuse with a size error that reads like a bug.

--size set slightly below the raw disk size at creation leaves room for this, and is what array vendors do.

Work it through

A file server has a four-disk RAID 5. Monday morning, /proc/mdstat reads:

md0 : active raid5 sdd1[3] sdc1[2] sdb1[1]
      5860532224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]

The server is up, users have no complaints. Reason it through before reading on.

Read the status first. [4/3], four members expected, three present. [_UUU]. The first is gone. sda1 is absent from the member list entirely, so it has not merely failed, it has been dropped.

How bad is this? RAID 5 survives one failure. It has had one. So the array is running with no remaining protection, and a second failure loses roughly 5.8 TB.

How long has it been like this? Nobody knows, which is the actual problem. sudo journalctl -k --since '7 days ago' | grep -i md and the SMART data on the remaining disks will give a timestamp. If the answer is “three weeks”, the monitoring is broken and that is a second incident.

What comes first? Not the disk swap. Verify the backup. The array is one failure from total loss, and a rebuild is the most stressful thing you can do to the surviving disks: it reads every sector of all three. If a second disk is marginal, the rebuild is exactly what will finish it off.

Then check the others before rebuilding. smartctl -a /dev/sdb on each survivor. Reallocated sectors or pending sectors on a second disk changes the plan entirely. You would then be copying the array off to somewhere else rather than rebuilding in place.

Then replace. --fail if not already, --remove, swap, --add, and watch. For 5.8 TB expect many hours, during which the array is still degraded.

Now the question underneath: why is a rebuild the riskiest moment in an array’s life? Because it is the only time every sector of every surviving disk is read. Disks accumulate unreadable sectors quietly; nothing finds them until something reads them. A rebuild reads all of them at once, on disks that are the same age and model as the one that just died, under sustained load.

That is the whole argument for RAID 6 on large disks, for scrubbing regularly so bad sectors surface while there is still redundancy, and for hot spares so the window between failure and rebuild is seconds rather than days.

And the practical habit: write down the monitoring check and test that it fires. This array degraded and told nobody. Every other decision here was downstream of that.

Try it

Optional, on a virtual machine with three spare disks. Genuinely do not do this on anything real.

1. sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc, then cat /proc/mdstat. Watch the initial resync.
2. sudo mdadm --detail /dev/md0. Compare Array Size to the disk sizes.
3. sudo mkfs.ext4 /dev/md0, mount it, write a file.
4. sudo mdadm --manage /dev/md0 --fail /dev/sdb, then read /proc/mdstat and your file. Note that nothing broke.
5. --remove it, --add the third disk, and watch the recovery bar.
6. Rebuild with --level=5 and three disks. Confirm the capacity is n−1.
7. sudo mdadm --monitor --scan --test and find out whether anything reaches you.

Verification step. You have it when you can glance at a /proc/mdstat line and say the level, how many members are missing, and whether another failure would be survivable.

Check yourself

References

• mdadm(8) - Linux man-pages project. Accessed 2026-08-07.
• md(4) - Linux man-pages project. Accessed 2026-08-07.
• mdadm.conf(5) - Linux man-pages project. Accessed 2026-08-07.

Every block above with a distribution and architecture header was captured by running the command on an AlmaLinux 10.2 container. Blocks without one are illustrative.