Deploy dark; release with a flag
Separating the act of shipping code from the act of turning it on makes the risky half of a change reversible in seconds without a deploy, and lets the two halves be judged by different people at different times.
Do this firstRolling back is one step, and it is practised
The code goes to production off. Someone turns it on afterwards, for some people, and can turn it off again without a build.
The reversibility gain is the obvious half. The one people underestimate is that it makes the release decision a separate, cheap, human decision — which is what lets a deploy pipeline run unattended without anyone feeling they have given up control. Teams that will not automate deploys often will automate them once turning a thing on is no longer the same event.
The cost is a flag system and the discipline to delete flags, which is underrated: a codebase of permanent flags is its own coupling problem.
The decoy
A staging environment. It answers “does this run”, which is rarely the question, and it answers it with traffic that does not resemble yours. It also costs about as much to operate as flags do to adopt.
Evidence
- Accelerate: State of DevOps 2019 — II self-reportedCluster analysis separates self-described delivery performance into distinct profiles, and continuous delivery capabilities — trunk-based development, automated testing, deployment automation, loosely coupled architecture — predict cluster membership in a PLS-SEM model.
- Development and Deployment at Facebook — IIIA concrete account of dark launch, flag-gated release and staged rollout operating together at scale: code live on all servers but hidden from users, a gating system deciding "which users see which features of the code" by criteria such as country or age, and rollout through internal-only, a few-thousand-machine stage, and full deployment — with rollback normally reverting a single commit.
- Google Cloud and Workspace global outage, 12 June 2025 — IIIThe inverse case: a code path shipped WITHOUT a flag sat globally live-but-dormant for about two weeks, and when an unrelated policy change activated it everywhere at once it crashed Service Control worldwide, taking down 50+ services. Google's own remediation was to make flags mandatory — to "enforce all changes to critical binaries to be feature flag protected and disabled by default".
Counterevidence
- In the Matter of Knight Capital Americas LLC — IIIA flag value that had been repurposed was safe on the seven servers that received the new deployment and catastrophic on the eighth, which did not: it re-activated dead code last used in 2003. SMARS "routed millions of orders into the market over a 45-minute period" and Knight "lost over $460 million from these unwanted positions". Flag reuse plus an inconsistent deployment produced an effectively irreversible blast radius.
- Feature Toggles: Practitioner Practices and a Case Study — IIIFlags accumulate as a maintenance liability even at a well-resourced organisation with a dedicated removal campaign: toggle count grew from 263 to 1,040 across the releases studied, developers made over 5,044 commits introducing or refactoring toggles, and of tracked release toggles only 20% were actually removed while 17% lingered in the source as debt.
- Holistic Configuration Management at Facebook — IIIConfiguration change is a first-class incident cause at scale: 16% of high-impact incidents over three months were configuration-related, split roughly 42% obvious errors, 36% subtle load-related issues, and 22% valid changes that exposed latent code bugs. Reports both a canary catching a log spew before rollout and a canary missing a latency regression because the tested scale was too small.
Last reviewed 2026-08-19.