Rolling back is one step, and it is practised
A rollback that is a single command and has been executed deliberately in the last quarter converts a bad deploy from an incident into an inconvenience; one that exists only on paper does not.
Do this firstOne command builds and tests the project from a clean checkout
One command, no coordination, no judgement calls about which pieces to revert. And — the part that is usually missing — executed on purpose, on a normal day, recently enough that someone remembers doing it.
An unrehearsed recovery path is a hypothesis. The interesting failures are never the command itself; they are the things around it that drifted since anyone looked, and the only way to find those is to use it while nothing is on fire.
The decoy
A documented rollback procedure. Documents are not rehearsed, and the first rehearsal will be during an incident at 3am by whoever is on call, who has never read it.
Evidence
- Development and Deployment at Facebook — IIIA concrete account of dark launch, flag-gated release and staged rollout operating together at scale: code live on all servers but hidden from users, a gating system deciding "which users see which features of the code" by criteria such as country or age, and rollout through internal-only, a few-thousand-machine stage, and full deployment — with rollback normally reverting a single commit.
- In the Matter of Knight Capital Americas LLC — IIIA flag value that had been repurposed was safe on the seven servers that received the new deployment and catastrophic on the eighth, which did not: it re-activated dead code last used in 2003. SMARS "routed millions of orders into the market over a 45-minute period" and Knight "lost over $460 million from these unwanted positions". Flag reuse plus an inconsistent deployment produced an effectively irreversible blast radius.
- Holistic Configuration Management at Facebook — IIIConfiguration change is a first-class incident cause at scale: 16% of high-impact incidents over three months were configuration-related, split roughly 42% obvious errors, 36% subtle load-related issues, and 22% valid changes that exposed latent code bugs. Reports both a canary catching a log spew before rollout and a canary missing a latency regression because the tested scale was too small.
Seen in the wild
Anonymised field observation. Illustration, not evidence — these carry no tier and cannot raise one.
A service came back from a restart healthy and empty. It passed its health check, served every request successfully, and had turned every feature off. The rollback worked; the verification did not.
Last reviewed 2026-08-19.