Data Nexus

State and recovery

How much stops working when one thing stops working — a property you design in, and discover the value of exactly once.

Also called Blast radius · Изоляция отказов

01/What it means

Every system has dependencies that are not essential to every operation, and the default is that all of them can take everything down. The recommendation service is slow, so the product page hangs, so the checkout that never called it times out too. Nobody designed that path; it is what happens when a call has no timeout and no fallback.

The controls are old and boring. A timeout on every outbound call, shorter than the patience of whatever is waiting. A fallback that degrades rather than fails — the page without recommendations rather than no page. A circuit that opens after repeated failures so the sick dependency is not also being hammered. And a separation of the paths that earn money from the paths that decorate them.

The test for whether any of this exists is not a document. Turn a dependency off in a rehearsal and see what stops. Teams are reliably wrong about the answer, and the direction of the error is always the same: more stops than expected.

02/What people get wrong

One slow dependency consumes every connection or thread in a common pool, and requests that never touch it queue behind those that do. The symptom is a whole platform degrading over minutes for a reason nobody can locate, because the failing component is not the one showing errors. Give the paths that matter their own budget, and let the decorative ones exhaust theirs alone.

How to test it

Knowing the definition is not the same as being able to check the figure. These are the procedures that do the second thing.

Testing a claim that you are covered
“We have backups. Everything is backed up nightly.” · 40 minutes, 6 questions.
Next

The definitions are the easy part. Whether the figure on your dashboard was computed this way is a different question, and usually the more expensive one.