Failure isolation

Every system has dependencies that are not essential to every operation, and the default is that all of them can take everything down. The recommendation service is slow, so the product page hangs, so the checkout that never called it times out too. Nobody designed that path; it is what happens when a call has no timeout and no fallback.

The controls are old and boring. A timeout on every outbound call, shorter than the patience of whatever is waiting. A fallback that degrades rather than fails — the page without recommendations rather than no page. A circuit that opens after repeated failures so the sick dependency is not also being hammered. And a separation of the paths that earn money from the paths that decorate them.

The test for whether any of this exists is not a document. Turn a dependency off in a rehearsal and see what stops. Teams are reliably wrong about the answer, and the direction of the error is always the same: more stops than expected.

A shared pool that every call draws from

One slow dependency consumes every connection or thread in a common pool, and requests that never touch it queue behind those that do. The symptom is a whole platform degrading over minutes for a reason nobody can locate, because the failing component is not the one showing errors. Give the paths that matter their own budget, and let the decorative ones exhaust theirs alone.

A definition sits still. This does not.

The term is a reading taken at a point in a sequence that is already running. These are the sequences, with the place each of them jams.

From something breaking to somebody confirming it works

An outage, from the moment the system stopped being correct to the moment a person has checked that it is correct again — including the parts that happen before anybody has touched a keyboard. 6 steps, monitoring whether the site is up is the arrangement that fails while looking reasonable.

Somebody will quote this number at you.

Knowing the definition is not the same as being able to check the figure. These are the procedures that do the second thing.

Testing a claim that you are covered

“We have backups. Everything is backed up nightly.” · 40 minutes, 6 questions.

Somebody is accountable for this number.

A definition is free. Being answerable for the figure it produces is the part that is bought, and this term is a working part of the engagements below.

Platform and transactional architecture

The domain model is the product. Everything above it is an interface to it.

---

Anti-crisis engineering and recovery

Stabilise before you improve. A system under stress cannot absorb a redesign.

Transaction

A change that either happens completely or does not happen at all — and the argument is always about where it ends.

---

Consistency

Whether two people looking at the system at the same moment see the same thing — and how long “eventually” is allowed to be.

---

RTO

How long you have decided the business can be down — and the only version of the number worth having is one somebody has timed.

---

Recoverable copy

A copy somebody has restored from. Until then it is a hypothesis, and the word backup is what the hypothesis is called before it is tested.

All ninety-six termsWhat this is regulated by in the UAE

Is this one right in your own numbers?

The definitions are the easy part. Whether the figure on your dashboard was computed this way is a different question, and usually the more expensive one.