State and recovery
RPO
How much recent work you have decided you can afford to lose — expressed in time, and settled by the business rather than by the infrastructure.
Also called Recovery point objective · Целевая точка восстановления
Formula
RPO ≈ Interval between recoverable copies
- Interval
- the time between one copy you have restored from and the next
- ≈
- an upper bound in the worst case: a failure the moment before the next copy loses the whole interval
Stated as an objective and measured against reality. Nightly copies mean an RPO of twenty-four hours whatever the policy document says, and the objective is only real when the interval and a tested restore both support it.
It is the cheaper of the two objectives to improve and the one more often left alone, because shortening the interval is a cost that recurs while the loss is a cost that has not happened yet.
It is also the one a business can answer without any technical vocabulary. Ask how many hours of orders, records or messages the company could re-enter by hand before the day stops being recoverable. That number is the RPO, and everything in the infrastructure is downstream of it.
- Copies taken
- nightly at 02:00
- Failure occurs
- 01:45
- Work since the last recoverable copy
- 23 hours 45 minutes
- Orders taken in a typical day
- 180
- Orders unrecoverable in the worst case
- ≈178
An RPO of twenty-four hours is not a technical parameter, it is a decision to re-enter or forfeit a day of business. Written down that way it is usually revisited; written as “nightly backups” it never is. The figures here are illustrative and the arithmetic is not.
The interval gets chosen for operational convenience and the business inherits it without ever being asked. Reverse the order: get the answer to how much work may be lost, then buy the interval that meets it. In most companies the honest answer is far smaller than nightly, and discovering that after a failure is the most expensive way to find out.
The ratios a decision is taken on. They inherit every error below them, which is why they are the last thing to trust and the first thing quoted.
Cannot be computed without
- Recoverable copy
- A copy somebody has restored from. Until then it is a hypothesis, and the word backup is what the hypothesis is called before it is tested.
The term is a reading taken at a point in a sequence that is already running. These are the sequences, with the place each of them jams.
- From something breaking to somebody confirming it works
- An outage, from the moment the system stopped being correct to the moment a person has checked that it is correct again — including the parts that happen before anybody has touched a keyboard. 6 steps, monitoring whether the site is up is the arrangement that fails while looking reasonable.
Knowing the definition is not the same as being able to check the figure. These are the procedures that do the second thing.
- Testing a claim that you are covered
- “We have backups. Everything is backed up nightly.” · 40 minutes, 6 questions.
A definition is free. Being answerable for the figure it produces is the part that is bought, and this term is a working part of the engagements below.
- Anti-crisis engineering and recovery
- Stabilise before you improve. A system under stress cannot absorb a redesign.
The definitions are the easy part. Whether the figure on your dashboard was computed this way is a different question, and usually the more expensive one.