I Depended on You and You Broke Me: An Empirical Study of Manifesting Breaking Changes in Client Packages
https://arxiv.org/abs/2301.04563
Method
"gathered all the metadata files (i.e., package.json files) from the published packages in the npm registry between December 20, 2010 and April 01, 2020"
Population
"a random sample of 384 client packages (95% confidence, 5% interval) covering 3,230 client releases and 5,957 test executions, against 1,909 provider packages"
What it does not show
npm specific. Test-based detection misses breaking changes the client’s own suite does not cover, so this undercounts. Does not measure what happened in the roughly 61% of unrecovered cases.
Suhaib Mujahid, Diego Elias Costa, Rabe Abdalkareem, Emad Shihab and others
11.7% of client packages and 13.9% of their releases were hit by a manifesting breaking change during non-major dependency updates, and 44% of those arrived in minor and patch releases — the versions the contract says are safe. Clients recovered in only 39.1% of cases.
Tier II: Large-scale registry mining combined with automated test execution over a statistically sampled cohort of real client packages and releases. Observational, no comparison of mitigation strategies.