De-Flake Your Tests: Automatically Locating Root Causes of Flaky Tests in Code At Google
https://storage.googleapis.com/gweb-research2023-media/pubtools/6478.pdf
Method
"In this paper, we present a novel technique to automatically identify the locations of the root causes of flaky tests on the code level to help developers debug and fix them."
Population
"We study the technique on flaky tests across 428 projects at Google." "We showed a notification for 8182 code changes to 79 unique developers."
What it does not show
The 82% rests on 83 manually inspected issues, not the full 428-project corpus. Developer-survey portions had low response rates and the paper flags responder bias. The cross-company rates are cited secondhand.
Celal Ziftci, Jim Cavalcanti
Root-cause localisation for flaky tests reached 82% accuracy on 83 manually verified historical issues. Cites comparable flaky-failure rates elsewhere — 4.56% at Google over 15 months, around 5% at Microsoft, 13% on TravisCI — suggesting a few percent of failures being flaky is a durable cross-organisation phenomenon rather than one company’s artefact.
Tier II: Peer-reviewed paper reporting deployment across 428 real projects with accuracy quantified against manually verified ground truth; observational, no randomised control.