If you inject a fault, hold back a control group
Fault injection produces evidence only when a randomly assigned control group receives the same traffic without the fault and a pre-declared metric is compared between the two; the design exists and is deployed, and its outcomes have not been published in quantified form by anyone.
Do this firstTurn it on for a cohort before everyone
The strongest thing available is a genuine randomised design: 1% of live users routed to a fault-injected canary, 1% to an untouched baseline, with statistical comparison of a declared metric before anything proceeds. That is a real experiment, running in production, on real traffic.
And the published outcome from it is the phrase “several cases where timeouts were set incorrectly”. No counts, no rates, no before-and-after.
This page exists because the gap is worth naming rather than papering over. The practice has a rigorous methodology, an enthusiastic literature, a decade of adoption, and — as far as an extended search could establish — not one published figure showing it changes an incident rate. Adopt it for the design, which is sound. Do not cite it as evidence, because there is none to cite.
The decoy
Chaos engineering as usually practised. Turn something off, watch dashboards, conclude the system is resilient. Without a control group there is nothing to compare the observation against, and the founding paper of the discipline describes the comparison design and then reports no results from running it — its one worked example is explicitly hypothetical.
Evidence
- Automating Chaos Experiments in Production — IVA randomised design for fault injection in production: users are assigned to a treatment group routed to a fault-injected canary and a control group routed to an untouched baseline, with a declared metric compared statistically between them. Reported outcome is qualitative — several cases where timeouts were set incorrectly and fallbacks revealed a service to be more business-critical than its owner intended.
What would settle it
Any measured comparison at adoption level: incident rate or recovery time before and after a team adopted fault injection, or between teams that did and did not. A 96-source review of the literature concludes the field is dominated by unvalidated solution proposals with limited empirical research, and this entry stays contested until someone publishes a number.
Last reviewed 2026-08-19.