Determinism · Feedback latency
A quarantined test has a deadline, not a drawer
Quarantining a flaky test with an expiry that fails the build converts an indefinite exclusion into a scheduled decision, which is the only mechanism that has been observed to make quarantine lists shrink.
Do this firstEvery change runs the suite before a human looks at it
Flaky tests must be removable from the critical path; that part is not controversial. What decides whether your suite recovers is what happens next.
A quarantine with an expiry date that fails the build when it passes forces the question back onto a human on a known day. Without one, the list only grows, because removing a test from quarantine requires someone to volunteer for an unbounded debugging session and nothing ever makes that the most urgent thing.
The decoy
A skip annotation with a ticket number in the comment. The ticket ages, the annotation does not, and in two years nobody can say whether the test still describes the system.
Evidence
- De-Flake Your Tests: Automatically Locating Root Causes of Flaky Tests in Code At Google — IIRoot-cause localisation for flaky tests reached 82% accuracy on 83 manually verified historical issues. Cites comparable flaky-failure rates elsewhere — 4.56% at Google over 15 months, around 5% at Microsoft, 13% on TravisCI — suggesting a few percent of failures being flaky is a durable cross-organisation phenomenon rather than one company's artefact.
- Flaky Tests at Google and How We Mitigate Them — IIIAt scale roughly 1.5% of test runs report a flaky result, almost 16% of tests show some flakiness at some point, and about 84% of the pass-to-fail transitions seen in CI are flakiness rather than real regressions.
- An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software — IIBuild failure is not a rare event: a mean failure ratio of 37% across 14 mature Java projects. Failures cluster — for 10 of the 14 projects, more than half and up to 80% of failed builds directly follow another failed build, and the fail ratio of the last 10 builds significantly predicts the next outcome in every project studied.
Seen in the wild
Anonymised field observation. Illustration, not evidence — these carry no tier and cannot raise one.
A test quarantined “until the feature ships” had never run. The suite used its recorded duration to decide what to parallelise, so its timing data — pure fiction — was shaping every other run.
Last reviewed 2026-08-19.