Gang of Eight: A Defect Taxonomy for Infrastructure as Code Scripts
https://akondrahman.github.io/files/papers/icse20_acid.pdf
Method
"We develop a taxonomy of IaC defects by applying qualitative analysis on 1,448 defect-related commits collected from open source software (OSS) repositories of the Openstack organization. We conduct a survey with 66 practitioners to assess if they agree with the identified defect categories included in our taxonomy."
Population
"80,425 commits collected from 291 OSS repositories spanning across 2005 to 2019"
What it does not show
Correlational, and drawn from a single organisation’s repositories. The outage examples the paper cites as motivation are anecdotes outside the measured corpus; the study does not quantify how often each category caused a production incident.
Akond Rahman, Effat Farhana, Chris Parnin, Laurie Williams
An eight-category defect taxonomy including one with no application-code analogue — idempotency, “defects that lead to incorrect system provisioning when the same IaC script is executed multiple times” — which surveyed practitioners recognised most strongly. The most frequent category overall was erroneous configuration data, not wrong logic.
Tier II: Observational mining of real defect-fixing commits cross-checked with a practitioner survey; frequency counts at scale, no control group, no production incident outcomes.