How to Fight Production Incidents? An Empirical Study on a Large-scale Cloud Service
https://acmsocc.org/2022/assets/slides/95.pdf
Method
"Dataset split: taxonomy (60 incidents); validation (30 incidents); test set (62 incidents). For each of the 6 dimensions: Populate summarized text from incident summary and post-mortem reports. Individually labels categories on taxonomy set. Identify common taxonomy via discussion."
Population
"Incidents from one year period (05/15/2021 to 05/15/2022)... Microsoft Teams service... a feature-blocker or outage incident (high severity)... incident has been resolved/mitigated... contains detailed root cause information"
What it does not show
Single service in a single organisation. Categories were assigned via a taxonomy the authors built and iterated themselves, so the boundaries reflect their judgement. Retrospective and associational: finding that most code-bug incidents had no monitor does not establish that adding one would have caught that incident.
Supriyo Ghosh, Manish Shetty, Chetan Bansal, Suman Nath
Quantifies the detection gap in a real large service: “≈17% of incidents either lacked monitors or telemetry coverage” and “10% incidents were not detected due to bugs, e.g., high threshold, buggy feature, wrong configuration”. Correlating cause with detection: “70% incident with code bugs does not have monitors”.
Tier II: Observational study of real incident records with a systematic manual coding taxonomy, no control group. Sourcing note: the full paper returned 403 everywhere tried; quotes are taken from the authors’ own official SoCC '22 slide deck, disclosed rather than reconstructed.