Flow · 6 steps · 6 known jams
An outage, from the moment the system stopped being correct to the moment a person has checked that it is correct again — including the parts that happen before anybody has touched a keyboard.
Finished when
The service is running, the data is right, somebody has verified both against something other than the dashboard, and the interval it was wrong for is a number the business has seen.
Done when The system has stopped being correct. Nobody knows yet, and this is where the clock starts whatever anybody later reports.
Where it stops
The incident timeline begins when somebody noticed.
It is the only moment anybody recorded. Reports that start at detection omit the most expensive term and make every recovery look faster than it was.
Cost. The business is told a two-hour recovery and lived through eleven. The gap is not dishonesty, it is a measurement that starts in the wrong place.
Done when A person who can act knows, and knows what broke rather than that something did.
Where it stops
Customers report it first.
Monitoring watches whether the service responds, not whether it is right. A site serving a stale build, a job failing silently, a form posting into nothing — all return 200 and all are broken.
Cost. The largest term in most real recoveries and the one absent from every estimate. A Friday-night failure found on Monday has spent three days before anybody has done anything.
Done when Somebody with the authority to accept the consequences has said restore, and the person doing it knows which copy.
Where it stops
An hour of messages establishing who can approve this.
The authority was never named, so out of hours it is a phone tree assembled under pressure by people guessing at the blast radius.
Cost. Pure elapsed time, spent while the failure continues, and entirely preventable by writing one name and one number down in advance.
Done when Data and service are back from a copy somebody has previously restored from.
Where it stops
The restore fails, or produces less than expected.
The copy has never been tested. Truncated dumps, tables dropped from the job when they were renamed, an environment that no longer exists to receive it, a key held only in the system that was lost.
Cost. This is where a recovery becomes a reconstruction, and the difference is measured in weeks rather than hours.
Done when Somebody has checked the state is correct, against records outside the system that broke.
Where it stops
Service is declared restored because it responds.
Verification is confused with availability. A platform can be up and serving yesterday's prices, a partial catalogue or an inconsistent ledger, and every one of those is a second incident starting quietly.
Cost. The damage continues under the impression it has stopped, which is worse than the outage — during an outage nobody transacts.
Done when The interval, the loss and the cause are written down, and the four terms are separated so the expensive one is visible.
Where it stops
The post-mortem names a technical cause and closes.
The cause is easier to discuss than the timeline, and fixing the cause prevents one failure while fixing the timeline shortens all of them.
Cost. The next incident of a different kind runs exactly as long as this one did.
It is the check everybody has, it is cheap, and it catches the failure mode that matters least. A service that has stopped responding announces itself; the expensive failures all return 200 — a stale build, a silent job, a form posting into nothing, a price list that stopped updating. Uptime monitoring reports green through all of them, which is worse than no monitoring, because it produces confidence.
InsteadMonitor a claim rather than a response. Is what is live what we last shipped. Did the nightly job write the rows it should have. Does the total on this page reconcile with the ledger. Each is a sentence that can be false while the service is perfectly available, and each is the sentence somebody would have wanted checked.
Every process has a written form and an actual one, and the queue is always in the actual one. We map what happens, from the records rather than from the meeting.