Automating Chaos Experiments in Production
https://arxiv.org/abs/1905.04648
Method
"a statistical analysis of metrics collected from a canary cluster and compares it to a baseline cluster, in order to determine whether there has been a statistically significant impact on any metrics of interest"
Population
"1% of active users to be in the experiment (treatment) group"
What it does not show
No aggregate quantified outcome: no share of experiments that found a regression, no before-and-after incident rate, no recovery-time comparison. The randomisation is per-experiment over users, not a comparison of teams or services that did and did not adopt the practice, so it cannot support any claim that the practice reduces incidents.
Ali Basiri, Lorin Hochstein, Nora Jones, Haley Tucker
A randomised design for fault injection in production: users are assigned to a treatment group routed to a fault-injected canary and a control group routed to an untouched baseline, with a declared metric compared statistically between them. Reported outcome is qualitative — several cases where timeouts were set incorrectly and fallbacks revealed a service to be more business-critical than its owner intended.
Tier IV: The design is a genuine randomised treatment-and-control experiment on live traffic — but the paper reports no quantified outcome from running it, only qualitative examples. Graded on what it reports rather than on what it could have reported.