An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software
https://dsg.tuwien.ac.at/team/trausch/pub/PID4727015.pdf
Method
"The study is conducted using real-world data gathered from 14 OSS projects that employ CI using GitHub and Travis-CI... The build metadata are gathered by querying the Travis-CI REST API using a custom-built crawler... Logfiles of builds are crawled from Amazon S3, the storage backend used by Travis-CI"
Population
"We analyzed a total of 54 248 logfiles, 92% of which were successfully assigned an error category... the analyzed projects exhibit a mean failure ratio of 37%"
What it does not show
Correlational only, by the authors’ own statement. Restricted to Java projects on Travis-CI with Maven or Gradle; deliberately sampled high-profile projects, so it may not generalise to smaller ones. Does not separate flaky failures from real ones, which is the distinction the quarantine control turns on.
Thomas Rausch, Waldemar Hummer, Philipp Leitner, Stefan Schulte
Build failure is not a rare event: a mean failure ratio of 37% across 14 mature Java projects. Failures cluster — for 10 of the 14 projects, more than half and up to 80% of failed builds directly follow another failed build, and the fail ratio of the last 10 builds significantly predicts the next outcome in every project studied.
Tier II: Large-N observational study of real CI builds across 14 projects over eight months, correlational only — the authors state the methodology ‘does not allow us to establish causal relationships’.