Reproducible Builds: Increasing the Integrity of Software Supply Chains
https://arxiv.org/abs/2104.06020
Method
"they developed a continuous integration (CI) system which builds each package in the Debian archive twice in a row, using two independent build environments that are deliberately configured to differ as much as possible. For instance, the clock on the second build is set 18 months in the future, and the hostname, language, system kernel, etc., are all varied so that if any environmental differences are used as a build input, the two builds will differ as a result."
Population
"Seven years later, over 95% of the 30 000+ packages in Debian's development branch can now be built reproducibly, and as the Linux distribution with the largest total number of reproducible packages, it serves as an extremely relevant case study."
What it does not show
No controlled comparison against non-reproducible practice: it does not measure a defect or incident rate difference between organisations that pin and hermeticise and those that do not. The 95% is specific to Debian’s ecosystem and tooling. It is evidence that determinism is attainable, not evidence of what it buys you.
Chris Lamb, Stefano Zacchiroli
Bit-for-bit identical builds are achievable at scale: over 95% of 30,000+ packages, verified by deliberately varying about thirty environmental parameters between two builds and diffing the result. Names the concrete non-hermetic causes found this way — embedded timestamps, filesystem ordering, locale-dependent hashing, build paths.
Tier II: Observational measurement of a live 30,000+ package distribution by a continuous integration system running since 2014. Real production data at scale, no treatment/control comparison.