Before you read. A substation fault takes out power to an office at 09:40 on a Tuesday. The generator starts. Forty minutes later it stops, because the fuel contract lapsed in 2023.
The finance system was last backed up at 02:00. Nobody knows how long it takes to bring it up somewhere else, because nobody has done it.
How much work has been lost, and how long will the business be down? Notice that those are two separate questions with two separate answers.
This topic is about two numbers and what they cost. It reads as management vocabulary and it is not: both numbers are decisions somebody in this job makes, usually by default, usually without anybody noticing a decision was made.
Some words you will need
- RPO
- Recovery point objective. How far back the last usable copy of the data may be.
- RTO
- Recovery time objective. How long the service may be unavailable.
- MTBF
- Mean time between failures. How long a thing runs before it breaks.
- MTTR
- Mean time to repair. How long it takes to fix once it has.
- failover
- Moving service from the thing that broke to the thing that did not.
- tabletop exercise
- Walking through the plan in a room, without touching anything.
What breaks without this
Work disappears and nobody expected it to. Everything since the last backup is gone, and the number of hours that represents was never agreed with anybody.
A standby site turns out not to work. It was bought, it was paid for monthly, and the first time anybody tried to serve from it was during the incident.
The recovery takes as long as it takes. Which is a fine answer until somebody asks why the shop was shut for two days.
Two numbers measured in opposite directions
These two get taught as a pair and confused constantly, and the confusion is easy to remove by noticing that they point in different directions from the same moment.
The scenario at the top has an RPO of seven hours and forty minutes, because the last backup was at 02:00 and the failure was at 09:40. Nobody chose that. It is the consequence of a backup schedule somebody set years ago, and it is the answer to “how much work has been lost” whether or not anybody has ever said it out loud.
The RTO is unknown, which is worse than a bad number. An organisation that says “four hours” and misses it by two has a problem. One that has no figure at all cannot tell whether the recovery is going well.
A useful way to keep them apart. RPO is a decision about data, and its unit is work. RTO is a decision about money, and its unit is trading. Whenever you cannot remember which is which, ask whether the sentence is about what was lost or about how long it took.
If you already set these: why the numbers are a business decision that keeps arriving as a technical one
Both figures are frequently produced by whoever runs the infrastructure, and both are properly decisions for whoever carries the loss.
Asked directly, a business will say it can tolerate no data loss and no downtime, because the question sounds free. It stops being free when the arrangements are costed: near-zero data loss means synchronous replication and the write latency that comes with it, and near-zero downtime means running the second site hot. Presenting the number alongside its price turns an aspiration into a choice, and the number that comes back is usually far more relaxed than the first answer.
The second thing that gets skipped is that the figures differ by system. One recovery objective for the whole estate is either unaffordable, because it is set by the most critical thing, or inadequate, because it is set by the average. Tiering is the practical answer: a handful of systems with tight numbers, most with relaxed ones, and the tier recorded in the inventory alongside what each system is for.
Which is where this connects back to topic 36. A recovery plan needs to know what breaks if a given system is gone, and that is the inventory field automatic discovery cannot fill. Organisations that cannot tier their recovery objectives are usually not short of recovery expertise. They are short of anybody who can say what the systems do.
Two more numbers, about the equipment rather than the plan
MTBF is how long a thing runs before it breaks. It comes from the manufacturer, it is a statistical figure across a population rather than a promise about your unit, and it feeds into how much redundancy you buy.
MTTR is how long it takes to fix once it has broken. This one is yours rather than the manufacturer’s, and it is the number people under-estimate, because it includes noticing, finding the spare, getting somebody to site, and the part where the replacement needs a firmware version you do not have.
The relationship worth carrying: MTTR is part of RTO. If you have promised a four hour recovery time and your mean time to repair a failed switch is six hours because the spare is in another building, you have not promised anything.
If you already quote these: what the manufacturer's figure does and does not promise
The reliability figure is the one most often misread, and the misreading is understandable because the number is enormous.
A quoted mean time between failures of several hundred thousand hours does not mean a unit is expected to last that long. It is derived from a population: run a large number of units for a short period, count the failures, and extrapolate. It describes the failure rate during the flat part of the life curve, after early failures and before wear-out, and it says nothing about either end of that curve.
Which is why it is useful for one thing and useless for another. Useful: estimating how many failures a fleet of two hundred identical devices will produce in a year, which is a stocking and staffing question with a real answer. Useless: predicting whether the specific switch in that rack will fail this year, which the figure was never about.
The repair figure is more actionable and is the one people underestimate, because they time the repair rather than the outage. The clock starts when the thing broke, not when somebody noticed, and it includes detection, diagnosis, finding a spare, getting somebody to the site, and verifying afterwards. A four hour hardware contract does not deliver a four hour recovery, and the gap between those two numbers is where availability calculations quietly become fiction.
What is already switched on
The three kinds of standby site are usually taught as a list of definitions. They are better understood as one question asked three times.
Cold is space, power and a contract. Nothing is racked. Recovering means buying or moving hardware, installing everything and restoring from backup, and the honest unit is weeks.
Warm has the hardware and the software and does not have current data. Recovery is a restore and a cutover, and the honest unit is hours.
Hot is running now, with data kept current, and can take traffic. Minutes, and in some designs no interruption at all.
The middle one is where most organisations actually are, and where most of the surprises live, because “we have a warm site” describes a wide range of readiness. The useful question is not which word applies. It is which of those rows is true today, and how anybody would know.
The step nobody has taken
Once a second site exists, there is a further choice, and it is the one that decides whether the site works when you need it.
Active-passive puts everything on one site and keeps the other ready. It is simpler, it is cheaper, and it contains a failover.
Active-active runs both. Losing one is a capacity event rather than a procedure, and the standby is being tested continuously because it is not a standby.
The reason this matters more than it sounds is that a failover which has never run is a plan rather than a capability, and the two are easy to confuse on a diagram. Every organisation with an untested failover believes it has a second site. Some of them do.
If you already work on networks: what actually goes wrong during a failover, and why the network is usually the reason it took longer than the plan said
Recovery plans tend to be written by the people who run the applications, and they tend to assume the network is a constant. It is not, and the same handful of network problems turn up in incident reviews.
Addresses. If the standby site uses different addressing, then everything with an address written into it has to change: firewall rules, application configuration, allow lists at partners, monitoring, licences pinned to an address. Stretching a subnet between sites avoids that and introduces a layer 2 failure domain spanning both, which is a decision with its own consequences and should be made deliberately rather than discovered.
Names. The usual mechanism for moving traffic is DNS, which means the time to live on the record is a hard floor under the recovery time. A record with a twenty four hour TTL cannot move traffic in an hour, and lowering it during the incident does not help, because the old value is already cached. Lowering it in advance is free and is the single cheapest recovery time improvement available to most organisations.
Routing. Anycast and dynamic routing move traffic faster than DNS and need the addresses to be yours and the routing to be in place beforehand. This is the mechanism behind most of the recoveries measured in seconds rather than hours.
Capacity. The standby link is frequently sized for replication rather than for users, because that is all it has ever carried. Failing over then works perfectly and the site is unusable, which reads as a failure of the standby site and is arithmetic nobody did.
The direction of the dependency. Authentication, name resolution and time all have to work before anything else does, and all three frequently live at the primary site. A recovery plan that starts by restoring the application and discovers halfway through that nothing can authenticate has the order wrong, and the order is the part a tabletop exercise finds cheaply.
The pattern behind all five is the same. Each one is a thing that was true at the primary site for so long that it stopped being visible, and none of them appear on a diagram of the recovery design.
Testing the plan rather than the hardware
The exam separates two kinds of exercise and the distinction is worth having, because they find different faults.
A tabletop exercise is a discussion. The people who would run the recovery sit in a room and walk through it, and nothing is touched. It is cheap, it can be run in an afternoon, and what it finds is decisions and dependencies: who declares a disaster, who authorises the failover, what order things come up in, and which step nobody can explain.
A validation test actually does it. Failing over to the standby, running from it, and failing back. It is expensive and disruptive, and what it finds is reality: the firewall rule that was never created, the expired certificate, the service that does not start because it looks for a licence server that is at the other site.
Both are worth doing and they are not substitutes. A tabletop finds problems that a technical test would never surface, because it exposes the gaps between people rather than between systems. A validation test finds problems no discussion could, because nobody knows they exist.
And there is a third thing that is neither. A plan that names a person who left in 2022 has failed before the exercise starts. Contact details, escalation paths and who has the authority to declare a disaster all go stale faster than the technical content, and reviewing them costs an hour a year.
Prove it
Find your own RPO. Not the one in a document. Look at when the last backup of something important actually completed, and subtract that from now. That number is your current recovery point objective whether or not anybody chose it.
Then ask three people what the RTO is. If you get three answers, that is the finding. If you get three identical answers, ask what it is based on.
Look up one TTL. Find the DNS record that would have to change to move traffic to a standby site and read its time to live. That number is a floor under the recovery time and it is usually a surprise.
What trips people up
1. Swapping the two objectives
RPO looks backwards from the incident and is about data lost. RTO looks forwards and is about time down. Both start at the same moment.
2. Treating MTTR as somebody else’s number
Mean time between failures comes from the manufacturer. Mean time to repair is yours, it includes finding the spare and getting somebody to site, and it sits inside the recovery time you have promised.
3. Calling a site hot because it exists
Hot means running now with current data and able to take traffic. A site with hardware in it and a backup from last night is warm, and the difference is measured in hours of recovery.
4. Believing an untested failover
The failover step is the only part of an active-passive design that has never operated, which makes it the part most likely to fail.
5. Forgetting the network in the recovery plan
Addresses, DNS time to live, link capacity and the order that authentication and name resolution come up in are all network decisions, and all four routinely turn a planned recovery into a longer one.
6. Treating a tabletop and a validation test as the same exercise
One finds decisions and dependencies for the cost of an afternoon. The other finds configuration that does not exist, and costs an outage. Neither substitutes.
Work it through
The office with no power, a generator that ran out of fuel, and a backup from 02:00.
Start with the two numbers, because everything else follows from them and because they are answerable immediately. The recovery point is seven hours and forty minutes of work, and that is now a fact rather than a target. The recovery time is unknown, and the first useful action is to stop treating it as unknown: pick somebody to establish what bringing the finance system up elsewhere would actually involve, and put a number on it, even a rough one, because you cannot manage an outage against a blank.
Then separate the incident from the finding. The incident is that a building has no power. The finding is that a fuel contract lapsed three years ago and nothing noticed, which is a lifecycle failure of exactly the kind topic 37 described: a thing that stopped being true on a date and produced no symptom until the day it mattered. The generator is not the only item in that category, and the question worth asking afterwards is what else expired quietly.
For the immediate recovery, the order matters more than the speed. Authentication, name resolution and time have to be available before the finance system is of any use to anybody, and if those services are also in the dark building then restoring the application first is wasted effort. That ordering is the single most common thing a tabletop exercise finds, and this organisation has not had one.
On the objectives themselves, seven hours forty is either acceptable or it is not, and that is a decision for whoever owns the finance system rather than for the network team. What the network team owes them is the honest price list. Halving the recovery point means backing up twice as often. Getting it under an hour means something closer to continuous replication and a link sized to carry it. Those cost money, and the conversation only works if somebody has already established what a day of lost trading costs, which is the number that makes all the others decidable.
Finally, the thing that will not appear in the incident report. Everybody in that building assumed the generator meant the power problem was handled, and they were right for as long as the fuel lasted. A control that works until a condition changes, with nothing watching the condition, is a common shape, and it is worth looking for the others before the next Tuesday.
Try it
Write down the recovery point and recovery time for one service. Any service. The exercise is not the writing, it is finding out that two people disagree.
Walk one failover through on paper. Twenty minutes, out loud, with somebody who would be in the room. Note every step somebody has to guess at.
Check a fuel or maintenance contract. Any of the invisible dependencies. The question is not whether it exists, it is when it expires and who would find out.
Check yourself
A failure happens at 09:40 and the last backup completed at 02:00. Which objective does that describe, and what does the other one describe?
It describes the recovery point: seven hours and forty minutes of work exists only in the failed system, so that is how much has been lost. Recovery point is measured backwards from the incident and is a statement about data.
The recovery time is the other direction: how long from the failure until the service is usable again. It is a statement about how long the business is not trading, and it is bought by keeping capacity elsewhere rather than by backing up more often.
What separates a warm site from a hot one, and why is that the expensive row?
Current data. Both have hardware racked and software installed; a hot site also holds data that is up to date and can take traffic now.
It is the expensive row because everything above it is equipment somebody can buy once, while keeping data current needs a continuous connection, continuous capacity and continuous attention. It is also the row that decides both objectives at once, because it sets the recovery point as well as most of the recovery time.
Why is an active-active pair usually more trustworthy than an active-passive pair with the same equipment?
Because there is no untested step in it. Both sites carry users continuously, so the configuration on each is proved every day, and losing one becomes a capacity problem rather than a procedure that has to run correctly under pressure.
In active-passive the failover is the only part of the design that has never operated. Its failures are usually mundane: a firewall rule that permits one site and not the other, a licence pinned to a serial number, a DNS record with a long time to live.
A recovery plan targets a two hour recovery time and traffic is moved by changing a DNS record with a 24 hour time to live. What is wrong?
The time to live is a floor under the recovery time. Resolvers that have cached the old answer will keep using it for up to a day, so the change cannot take effect within two hours no matter how quickly the standby comes up.
Lowering the value during the incident does not help, because the cached copies already carry the old figure. Lowering it in advance costs nothing and is one of the cheapest recovery time improvements available.
What does a tabletop exercise find that a full validation test does not?
The gaps between people rather than between systems. Who declares a disaster, who authorises the failover, what order services have to come up in, and which step nobody present can explain.
It costs an afternoon and no outage, which is why it can be run often enough to keep the answers current. A validation test finds a different class of problem, the configuration that does not exist, and neither one substitutes for the other.
References
- NIST SP 800-34 Rev. 1 - NIST, the contingency planning guide, and the source of the recovery objective vocabulary. Free. Accessed 2026-08-12.
- NIST SP 800-84 - NIST, on test, training and exercise programmes, including the difference between a tabletop discussion and a functional test. Free. Accessed 2026-08-12.
- ISO 22301 - ISO, the business continuity management standard the commercial vocabulary comes from. Paid, and the site refuses automated requests, so the link checker reports it as forbidden rather than broken. Accessed 2026-08-12.
Pictures. A freely licensed file from Wikimedia Commons, downloaded and served from this site rather than linked across to somebody else’s server. Resized and otherwise unaltered.
- Data center backup generators by Rsparks3, CC0.
Where the output came from. Nothing on this page is captured, and there is no Across platforms section, because there is no command here to run on any of them. Disaster recovery is decisions and rehearsal, and a transcript of a decision is a screenshot of a meeting. The three figures carry what the prose would otherwise have to assert, which is the honest substitute when a topic has no observable behaviour to record.
If you also work on Linux. Backup and restore on the Linux+ track covers the mechanism underneath the recovery point objective: what gets copied, how often, and how you find out whether the copy is any good.