Backup and restore

Before you read. Ask an organisation whether it has backups and the answer is always yes. Ask when somebody last restored one and the answer is usually a pause.

Those are different questions, and only one of them matters, because the backup is not the deliverable. The restore is. Everything before it is preparation for an event that has not been rehearsed.

So: what could be wrong with a backup that runs every night, reports success, and has done for two years?

Quite a lot, and none of it is visible from the backup side. The archive could be empty. It could be missing the database because that file was locked. It could be missing SELinux contexts, so it restores and does not work. It could be fine and take eleven hours to restore, which for a four-hour outage target is the same as being broken.

Every one of those is discovered by restoring, and by nothing else.

Some words you will need

full backup

A complete copy of everything. Slow to make, simple to restore.

incremental

Only what changed since the last backup of any kind. Small, and a restore needs the whole chain.

differential

Everything changed since the last full. Grows daily, and a restore needs only two pieces.

RPO

Recovery point objective: how much data you can afford to lose. Decides backup frequency.

RTO

Recovery time objective: how long you can afford to be down. Decides backup type.

What breaks without this

The restore fails on the day it matters, which is the only day it is attempted.

You copy the wrong thing. rsync with and without a trailing slash do different things, and --delete pointed the wrong way removes the data you were protecting.

You trust something that is not a backup. RAID, snapshots, and replication all feel like safety and none of them survives the thing that actually happens, which is somebody deleting the wrong directory.

The three types

A week, concretely. Full on Sunday, then daily:

• Incremental: Monday holds Monday’s changes, Tuesday holds Tuesday’s. To restore Thursday you need Sunday, Monday, Tuesday, Wednesday, Thursday, five pieces, in order, all intact.
• Differential: Monday holds Monday, Tuesday holds Monday and Tuesday, Thursday holds everything since Sunday. To restore Thursday you need Sunday and Thursday. Two pieces.

The choice is RTO against storage. Incremental is cheapest to run and slowest and riskiest to restore. Differential costs more space each day and restores from two files. If nobody has stated an RTO, that conversation is the actual task and the backup design follows from it.

rsync

rsync copies a tree, and copies only what changed. That second part is why it beats cp for anything repeated: the second run transfers almost nothing.

rsync -av /srv/data/ /backup/data/

-a is the one to use by default. Without it you get files with today’s timestamps and your ownership, which makes the copy useless as a backup and useless for incremental comparison, because rsync decides what changed partly from timestamps.

The trailing slash

# Debian 13 (trixie), x86_64
$ cd /tmp; echo '--- without a trailing slash ---'; rsync -a source dest-a/; find dest-a; echo '--- with one ---'; rsync -a source/ dest-b/; find dest-b
--- without a trailing slash ---
dest-a
dest-a/source
dest-a/source/reports
dest-a/source/reports/q3.txt
dest-a/source/reports/q4.txt
dest-a/source/index.txt
--- with one ---
dest-b
dest-b/reports
dest-b/reports/q3.txt
dest-b/reports/q4.txt
dest-b/index.txt

--delete, safely

--delete makes the destination match the source exactly, which means removing anything the source no longer has. The destination below contains old.txt, which the source does not.

# Debian 13 (trixie), x86_64
$ cd /tmp; printf 'stale\n' > dest-a/old.txt; echo '--- what WOULD --delete remove? ---'; rsync -a --delete --dry-run --itemize-changes source/ dest-a/; echo '--- nothing has changed yet ---'; ls dest-a
--- what WOULD --delete remove? ---
*deleting   old.txt
>f+++++++++ index.txt
cd+++++++++ reports/
>f+++++++++ reports/q3.txt
>f+++++++++ reports/q4.txt
--- nothing has changed yet ---
old.txt

--dry-run with --itemize-changes is the whole safety story. It lists *deleting old.txt before anything happens, and ls afterwards confirms nothing did.

The itemize codes are worth a glance: >f means a file is being sent, cd means a directory is being created, *deleting is self-explanatory. The +++++++++ means every attribute is new, so this is a first copy.

--delete makes the destination match the source, which is what you want for a mirror and is dangerous in exactly one situation: if the source is empty or unmounted, “match the source” means “delete everything”. A backup job whose source is an unmounted NFS share will faithfully empty the backup.

Two protections worth having: --max-delete=100, which aborts if the run would remove more than expected, and checking that the source is mounted before the job runs at all.

rsync -a --delete --link-dest=/backup/2026-08-06 /srv/data/ /backup/2026-08-07/

psql -c "SELECT pg_backup_start('nightly');"
lvcreate -s -n dbsnap -L 20G /dev/vg0/dbdata
psql -c "SELECT pg_backup_stop();"
mount -o ro /dev/vg0/dbsnap /mnt/snap && rsync -a /mnt/snap/ /backup/

dd, and block-level copies

dd copies bytes, knowing nothing about files or filesystems. That makes it the tool for whole devices, boot sectors, and disk images, and a poor tool for anything rsync can do.

# AlmaLinux 10.2, aarch64
$ echo '--- dd reports what it did ---'; dd if=/dev/zero of=$DEV0 bs=1M count=64
--- dd reports what it did ---
64+0 records in
64+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 0.0590037 s, 1.1 GB/s

if= input file, of= output file, bs= block size, count= how many blocks.

bs matters for speed. The default is 512 bytes, which means one system call per half-kilobyte. bs=4M is dramatically faster on any real transfer.

status=progress prints a running total, which for a multi-hour disk clone is the difference between watching a progress figure and wondering whether it has hung.

And the verification that makes it a backup rather than a hope:

# AlmaLinux 10.2, aarch64
$ mkfs.ext4 -q -L payroll $DEV0; echo '--- clone the whole device ---'; dd if=$DEV0 of=$DEV1 bs=4M status=none; echo '--- do the two match? ---'; sha256sum $DEV0 $DEV1
--- clone the whole device ---
--- do the two match? ---
b92836ae943988944dcf3daa909c1078515f6ba1c5d599106a69e36439aac002  /dev/loop0
b92836ae943988944dcf3daa909c1078515f6ba1c5d599106a69e36439aac002  /dev/loop1

Identical checksums, so the copy is provably byte-for-byte. That is the shape of every verification in this lesson: produce a number from the original, produce a number from the copy, compare. Anything less is an assumption.

dd has no undo and no confirmation. if= and of= are two characters apart and reversing them overwrites the source with the destination. The old nickname is “disk destroyer” and it is earned. Read the line twice, and prefer lsblk to confirm the device names immediately before.

ddrescue rather than dd for a failing disk. dd stops or stalls on a read error; ddrescue skips, records what it could not read in a map file, and comes back for the bad regions afterwards with retries. On a dying disk that difference is most of your data.

lvcreate -s -n db-snap -L 20G /dev/vg/db
mount -o ro /dev/vg/db-snap /mnt/snap
# back up /mnt/snap, then
lvremove -f /dev/vg/db-snap

What is not a backup

This deserves saying plainly, because all three feel like protection.

RAID is not a backup. It replicates writes, so a deletion is replicated instantly to every member. It protects against a disk failing and nothing else, which is the point made at length in lesson 15.

Snapshots are not backups. They live on the same storage as the original, so they do not survive the array failing, the machine being stolen, or the filesystem corrupting. They are excellent for “I deleted that an hour ago” and useless for “the data centre flooded”.

Replication is not a backup. It is RAID over a longer wire. Corruption replicates faithfully and quickly.

The framing that survives: 3-2-1. Three copies of the data, on two different kinds of media, one of them offsite. The offsite copy is the one that answers the question the other two cannot.

Worth adding a modern fourth: one copy the production system cannot delete. Ransomware that takes the machine takes everything the machine can write to, which includes the backup share it has credentials for. Immutable or write-once storage, or a pull-based backup where the backup server reaches in rather than the host pushing out.

find /srv/data -type f | wc -l
find /restore/data -type f | wc -l
cd /srv/data && find . -type f -exec sha256sum {} + | sort > /tmp/orig.sha
cd /restore/data && find . -type f -exec sha256sum {} + | sort > /tmp/rest.sha
diff /tmp/orig.sha /tmp/rest.sha

Across distributions

The tools are the same everywhere; only the packaging differs.

gddrescue on Debian is a real trap: the package is gddrescue and the command is ddrescue, and there is a different, older, worse tool called dd_rescue with an underscore. Install gddrescue, run ddrescue.

For anything beyond a single machine, restic and borgbackup are what most people should reach for now: deduplicating, encrypted, verifiable, and with a real check command. Neither is on the objective list and both are worth knowing.

Prove it

Before trusting any backup:

# Does the archive contain what you think
tar -tzf backup.tar.gz | wc -l
tar -tzf backup.tar.gz | grep -c 'important/'

# Is it intact
gzip -t backup.tar.gz && echo "checksum ok"

# Restore it somewhere harmless and compare
mkdir /tmp/verify && tar -xzf backup.tar.gz -C /tmp/verify
diff -r /srv/data /tmp/verify/srv/data | head

# And the one that counts
# ... start the service against the restored data and use it

The last line has no command because it is not one. Everything above proves bytes survived; only running the thing proves the backup is useful. Put it in the calendar, do it quarterly, write down how long it took.

What trips people up

1. The trailing slash

rsync -a source dest/ copies the directory; rsync -a source/ dest/ copies its contents. The source’s slash is what matters and the destination’s is ignored.

--dry-run the first time, every time. The failure is silent and it compounds nightly.

2. --delete pointed the wrong way

rsync -a --delete /backup/ /srv/data/ makes production match the backup, which is a restore. Swap the arguments and it makes the backup match production, which is a mirror. They look almost identical on the command line.

Always --dry-run first. --max-delete=100 limits the damage from the case nobody predicted.

3. A backup nobody has restored

Covered throughout. Two years of green ticks is evidence that a script exits zero.

4. dd with if and of swapped

There is no confirmation and no undo. lsblk immediately before, read the line twice, and be more careful than usual when tired.

5. Backing up a running database by copying its files

You get a file that was never in that state, which restores into corruption.

Use pg_dump or mysqldump, or snapshot the filesystem underneath, or stop the service. Copying a live database’s data directory is the classic backup that passes every test except the one that matters.

Work it through

A ransomware incident. A file server’s data is encrypted. You are asked what can be recovered.

The setup: RAID 6 on the server; hourly LVM snapshots kept for two days; nightly rsync to a NAS mounted at /mnt/backup with --delete; monthly full backup to tape, offsite, last taken eighteen days ago.

Work out what survives before reading on.

RAID 6 gives you nothing. It replicates writes. Every encrypted block was written to all the members faithfully and immediately. The array is perfectly healthy and perfectly encrypted.

The LVM snapshots may survive, and probably do not. They are on the same volume group, so anything with root on that machine can remove them, and ransomware routinely does exactly that, because it is a well-known recovery route. If they were missed, hourly snapshots give you an excellent recovery point. Check before assuming either way; this is the first thing to look at because it is the cheapest win.

The NAS is the painful one. It was mounted, the server had write access, and the nightly rsync ran with --delete. So either the ransomware encrypted it directly over the mount, or the backup job ran after the encryption and faithfully replicated the encrypted files, deleting the good ones to match. A mounted, writable backup destination is inside the blast radius.

The tape is the answer, and it is eighteen days old. So the recovery point is eighteen days of lost work, and the recovery time is however long a tape restore takes, which nobody has measured.

Now the design conclusions, which are the point of the exercise.

Offsite mattered and offline mattered more. The tape survived because the compromised machine could not reach it. That is the property doing the work, not the medium.

A mounted backup share is not a backup. Anything the production system can write to, an attacker on that system can write to. The fix is a pull model, the backup server reaches in over SSH with a restricted key, or immutable storage with a retention lock the client cannot override.

Retention decides the recovery point, and monthly is a policy decision nobody made deliberately. Eighteen days of loss is the direct consequence of a tape schedule that was probably set by how many tapes were in the budget.

And the number nobody has: how long does a full tape restore take? If it is three days, the RTO conversation should have happened before the incident, not during it.

The habit worth taking from all of this: ask what could delete the backup. Not what could fail, what could delete it. RAID, snapshots, and a mounted share all answer “the same thing that destroyed the original”, and that is the question that separates a copy from a backup.

Try it

Optional, on any machine. Nothing here touches real data if you use /tmp.

1. mkdir -p /tmp/src/sub && echo one > /tmp/src/a.txt && echo two > /tmp/src/sub/b.txt.
2. rsync -av /tmp/src /tmp/dest1/ then find /tmp/dest1. Note the extra level.
3. rsync -av /tmp/src/ /tmp/dest2/ then find /tmp/dest2. Compare.
4. echo stale > /tmp/dest2/old.txt, then rsync -a --delete --dry-run -i /tmp/src/ /tmp/dest2/. Read the *deleting line, and confirm the file is still there afterwards.
5. Run it without --dry-run and confirm it is gone.
6. dd if=/dev/urandom of=/tmp/orig.img bs=1M count=8, copy it with dd, then sha256sum both and compare.
7. Time a restore of something real, and write the number down.

Verification step. You have it when you can write an rsync command for an unfamiliar pair of paths, predict exactly where the files land, and prove it with --dry-run before running it for real.

Check yourself

References

• rsync(1) - Debian Project. Accessed 2026-08-07.
• dd(1) - Linux man-pages project. Accessed 2026-08-07.
• sha256sum(1) - Linux man-pages project. Accessed 2026-08-07.
• GNU ddrescue manual - GNU Project. Accessed 2026-08-07.
• rsnapshot - rsnapshot project. Accessed 2026-08-07.

Every block above with a distribution and architecture header was captured by running the command on a Debian 13 (trixie) container and an AlmaLinux 10.2 container. Blocks without one are illustrative.