Restore from backup on a normal day, before you have to
A recovery path that has never been executed is a hypothesis, and the mechanisms most likely to fail are the ones that report success without ever having been asked to produce a usable restore.
Do this firstRolling back is one step, and it is practised
Take a real backup and restore it, on a Tuesday, with the clock running and somebody timing it.
The incident that makes the case is not a story about carelessness. Every sensible mechanism was in place; what was missing was any occasion on which one of them had been asked to do its job. Recovery came down to a single volume snapshot taken six hours earlier and an eighteen-hour copy over a throttled link.
The contrasting case is instructive for what it credits. Two recoveries of very large user-data losses at another organisation are attributed by their own write-up to the fact that similar situations had been simulated many times beforehand — one of them tooled and rehearsed weeks earlier during a scheduled disaster exercise.
Be clear about the evidence: both are case studies, and no large-N measurement of drilled against undrilled recovery appears to exist. The mechanism is plausible and the counting has not been done.
The decoy
Backups that report success. One organisation held five independent mechanisms — periodic dumps, volume snapshots, cloud disk snapshots, streaming replication, and a staging copy — and when it mattered, four had failed silently or were never enabled. The dumps had been broken for months and the failure emails were being rejected by the mail configuration.
Evidence
- Data Integrity: What You Read Is What You Wrote (SRE Book, ch. 26) — IIITwo recoveries of large user-data losses, both credited by the write-up to prior rehearsal. One restored 99%+ of data within hours of the initial estimate because similar situations had been simulated many times. The other recovered 436,223 of about 600,000 wrongly deleted audio references — 161,000 were unrecoverable because they were deleted before ever being backed up — requiring 5,337 tapes recalled by truck, reaching 74% by day two, 99.95% by day three and seven days in total, with the tooling built and tested weeks earlier during a scheduled disaster exercise.
- Postmortem of database outage of January 31 — IIIRoughly 300 GB destroyed by a command run against the wrong host. Five redundancy mechanisms existed — periodic dumps to object storage, volume snapshots copied to staging, cloud disk snapshots, streaming replication, and a staging copy — and the write-up states the process of both finding and using backups failed completely. The dumps had been silently failing for months on a version mismatch, and their failure notifications were being rejected by mail policy. Recovery used a staging snapshot from six hours earlier, copied over a throttled link, taking about eighteen hours, with final loss estimated at 5,000 projects, 5,000 comments and about 700 users.
Last reviewed 2026-08-19.