Holistic Configuration Management at Facebook
https://sigops.org/s/conferences/sosp/2015/current/2015-Monterey/printable/008-tang.pdf
Method
"we manually analyzed the high-impact incidents during a three-month period" ... "A config is associated with a canary spec that describes how to automate testing the config in production. The spec defines multiple testing phases. For example, in phase 1, test on 20 servers; in phase 2, test in a full cluster with thousands of servers."
Population
"In 2014, thousands of engineers made live configuration updates to various parts of the site... the tools manage hundreds of thousands of configs (i.e., configuration files) from a central location, and distribute them to hundreds of thousands of servers and more than one billion mobile devices."
What it does not show
No before/after comparison of incident rates against a no-canary baseline, and no aggregate catch rate — only illustrative cases in both directions. The 16% is one organisation’s manual classification over a single three-month window. The authors’ own conclusion is that canary testing needed improvement, not that it was sufficient.
Chunqiang Tang, Thawan Kooburat, Pradeep Venkatachalam, Akshay Chander, Zhe Wen, Aravind Narayanan, Patrick Dowell, Robert Karl
Configuration change is a first-class incident cause at scale: 16% of high-impact incidents over three months were configuration-related, split roughly 42% obvious errors, 36% subtle load-related issues, and 22% valid changes that exposed latent code bugs. Reports both a canary catching a log spew before rollout and a canary missing a latency regression because the tested scale was too small.
Tier III: Peer-reviewed systems paper reporting one organisation’s operational numbers plus a manually coded three-month incident sample; no control group and no claim the sample is representative.