Systems
Anti-crisis engineering and recovery
Something is on fire: data is gone, transactions are inconsistent, or the platform has stopped being reliable enough to trade on. This is the work of stabilising first and correcting second.
A migration, a vendor or a failure took data with it, and the backups are not what you were told they were.
Payments and inventory disagree, and support is reconciling by hand.
The platform is up but you have stopped trusting it enough to run promotions.
A developer left and took the only understanding of the deployment with them.
- 01
Containment
Stop the bleeding before improving anything. That usually means freezing the failure mode, isolating the affected path, and putting a manual control in place that is honest about being manual.
- 02
Reconstruction
Rebuilding lost state from whatever remains — logs, third-party records, exports, caches, counterparties. On one recovery this meant reconstructing 8,500 SKUs programmatically rather than by hand.
- 03
Transactional integrity repair
Finding every place the system can end up in a state it should not be able to reach, and closing it. Reconciliation first, then the constraint that prevents a recurrence.
- 04
Making the next failure survivable
Backups that are restored on a schedule rather than assumed, isolation between components that should not be able to take each other down, and an incident path that does not depend on one person.
- 01
A stabilised platform, with the failure mode contained and documented
- 02
Recovered data with a written account of what was recovered, what was reconstructed and what is gone
- 03
Integrity constraints that prevent the same class of failure
- 04
A recovery plan that has actually been tested
Invariant
Stabilise before you improve. A system under stress cannot absorb a redesign.
- How quickly can you start?
- For genuine incidents, within days. Write to us with what broke and when; we will tell you honestly whether we are the right team before we quote anything.
- Can data actually be recovered after a total loss?
- Sometimes, and rarely from the place you expect. On one engagement a store was rebuilt to a working system in two months after complete data loss, with the catalogue reconstructed from surviving fragments and third-party records. It depends entirely on what traces the business left elsewhere.
- Do you take over long term afterwards?
- Only if it is the right answer. The normal end state is a stabilised system your own team runs, with us available for the architecture questions that follow.
Each one carries what it is good for, what it is proven on here, and the point at which it stops working. We take no vendor commissions on any of them.
- Shopify
- Hosted commerce: Shopify runs the checkout, the payment compliance and the uptime; the theme and the data model are yours.
- PostgreSQL
- A relational database with real transactions: either every part of a change lands or none of it does.
- Prisma
- Prisma is a schema definition language for relational databases that generates a typed client and emits versioned SQL migration files.
- Sentry
- Sentry captures unhandled errors and slow transactions from running software, groups them into issues, and ties each one to the release and the commit that introduced it.
- WordPress
- The PHP content management system behind 40.7 per cent of all websites, which assembles every page at request time from a core, a theme and an unbounded number of third-party plugins.
- Tilda
- A hosted website builder where pages are assembled by dragging pre-made blocks, and the platform serves the result from its own CDN.
- End-of-life runtimes
- A language runtime or framework version whose maintainers have stopped issuing security fixes — still running, still serving traffic, no longer receiving patches.
- SQL
- The language a relational database is actually configured in — tables, constraints, indexes and access policies.
- Shell
- The glue that runs on the server: deployment, backups, recovery and the small operations nobody wants to do by hand twice.
- From a brief to work somebody has accepted
- A piece of paid work, travelling from the sentence that started it to the moment the buyer agrees it is finished and can carry it without the supplier. 6 steps, 6 known jams.
- From something breaking to somebody confirming it works
- An outage, from the moment the system stopped being correct to the moment a person has checked that it is correct again — including the parts that happen before anybody has touched a keyboard. 6 steps, 6 known jams.
Data loss, a platform that will not come back up, a supplier who has stopped answering, a migration that failed in production. The first conversation is about triage rather than scope — what is recoverable, in what order, and what will be lost regardless. We answer this line the same day.