Flaky Tests at Google and How We Mitigate Them
https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html
Method
"We define a 'flaky' test result as a test that exhibits both a passing and a failing result with the same code."
Population
"Across our entire corpus of tests, we see a continual rate of about 1.5% of all test runs reporting a 'flaky' result."
What it does not show
No published methodology for any of the figures — no dataset, time window or statistical treatment. Single-author self-report on a company blog. Does not break flakiness down by cause, and does not measure whether quarantine or expiry policies reduce it.
John Micco
At scale roughly 1.5% of test runs report a flaky result, almost 16% of tests show some flakiness at some point, and about 84% of the pass-to-fail transitions seen in CI are flakiness rather than real regressions.
Tier III: Engineering blog post reporting company-wide numbers with no published methodology, dataset or peer review.