Automated Canary Analysis at Netflix with Kayenta
https://netflixtechblog.com/automated-canary-analysis-at-netflix-with-kayenta-3260bc7acc69
Method
"Mann-Whitney U test" used to "compute confidence intervals" between baseline and canary metrics, with the overall score defined as "the ratio of metrics classified as 'Pass' out of the total number of metrics."
Population
"Kayenta runs approximately 30% of our production canary judgments, which amounts to an average of 200 judgments per day."
What it does not show
No before/after comparison of incident or rollback rates attributable to adopting automated canary analysis, and no reported false-positive or false-negative rate for the judgments. Engineering self-report; does not quantify how many bad deploys it actually blocked.
Michael Graff, Chris Sanden
An automated canary analysis system running at real production scale — roughly 200 judgments a day — gating deployments by statistically comparing time-series metrics between a canary and a baseline cluster rather than by eye.
Tier III: Single-org engineering report with production usage statistics and a described statistical method; no control group, not peer reviewed.