State and recovery
RTO
How long you have decided the business can be down — and the only version of the number worth having is one somebody has timed.
Also called Recovery time objective · Целевое время восстановления
Formula
RTO = Detection + Decision + Restoration + Verification
- Detection
- from failure to somebody knowing, which is often the largest term and is never in the estimate
- Decision
- from knowing to somebody with authority saying restore — out of hours, this is a phone tree
- Restoration
- the part everyone measures: data back, service running
- Verification
- confirming it is correct rather than merely up, which is what prevents restoring the damage twice
Estimates usually contain only the third term. A rehearsal measures all four, and the difference between the estimate and the rehearsal is routinely a factor of several.
The distinction from RPO is worth keeping sharp: RPO is what you lose, RTO is how long you are stopped. They are bought separately and traded against each other — a warm standby buys hours off the RTO and does nothing for the RPO, more frequent copies do the reverse.
Detection deserves its own attention because it is the term nobody plans. A failure at 02:00 on a Friday discovered by a customer at 09:00 on Monday has already spent three days of RTO before anyone has done anything, and no amount of restoration speed recovers it. That is a monitoring decision wearing a recovery cost.
“We can restore in two hours” is a statement about one of four terms. The number that matters to the business starts when the system broke and ends when somebody has confirmed the service is correct. Rehearse it end to end, unannounced, including finding the person who is allowed to authorise it — and use the measured figure, because the estimate is always the optimistic term alone.
The ratios a decision is taken on. They inherit every error below them, which is why they are the last thing to trust and the first thing quoted.
Cannot be computed without
- Recoverable copy
- A copy somebody has restored from. Until then it is a hypothesis, and the word backup is what the hypothesis is called before it is tested.
The term is a reading taken at a point in a sequence that is already running. These are the sequences, with the place each of them jams.
- From something breaking to somebody confirming it works
- An outage, from the moment the system stopped being correct to the moment a person has checked that it is correct again — including the parts that happen before anybody has touched a keyboard. 6 steps, monitoring whether the site is up is the arrangement that fails while looking reasonable.
Knowing the definition is not the same as being able to check the figure. These are the procedures that do the second thing.
- Testing a claim that you are covered
- “We have backups. Everything is backed up nightly.” · 40 minutes, 6 questions.
A definition is free. Being answerable for the figure it produces is the part that is bought, and this term is a working part of the engagements below.
- Anti-crisis engineering and recovery
- Stabilise before you improve. A system under stress cannot absorb a redesign.
The definitions are the easy part. Whether the figure on your dashboard was computed this way is a different question, and usually the more expensive one.