State and recovery
Rebuilding what the system knew from whatever survived it — the discipline that exists because the copy you were counting on is the thing that failed.
Also called State rebuild · Реконструкция данных
When there is nothing to restore from, the state is rarely gone entirely; it is scattered. Logs, third-party records, exports somebody kept, caches, search indexes, the counterparties who hold their half of every transaction, and in commerce the suppliers whose catalogues you sell. Reconstruction is the work of establishing which of those is authoritative for which field and assembling a defensible version from them.
The order of operations matters more than the tooling. Identify an authoritative source per field before writing anything, because a reconstruction that merges two sources without a rule produces a dataset nobody can later reason about. Record the provenance of every reconstructed value. And reconstruct the invariants first — the constraints that must hold — so the rebuilt state cannot be internally contradictory even where it is incomplete.
This practice has done it at scale once: after a cyberattack destroyed a retailer's site, product database, pricing and every backup, 8,500 catalogue entries were rebuilt programmatically rather than by hand, from the sources that remained. It is the least glamorous engineering there is and the only thing that helps on the day.
A team facing a lost catalogue starts re-typing, because it feels like progress and the first hundred rows go quickly. At a few thousand rows it is months of work with a typo rate nobody is measuring, and no record of where any value came from. Establish sources and write the pipeline; it is slower for two days and finished in a fraction of the time, with provenance attached.
Knowing the definition is not the same as being able to check the figure. These are the procedures that do the second thing.
- Testing a claim that you are covered
- “We have backups. Everything is backed up nightly.” · 40 minutes, 6 questions.
The definitions are the easy part. Whether the figure on your dashboard was computed this way is a different question, and usually the more expensive one.