Tricorder: Building a Program Analysis Ecosystem
https://research.google.com/pubs/archive/43322.pdf
Method
"We include an empirical, in-situ evaluation of the tool as it is used by developers across Google that shows the usefulness and impact of the platform."
Population
"Every workday at Google, engineers perform more than 800k builds, run 100M test cases, produce 2PB of build outputs, and send 30k changelist snapshots (patch diffs) for review."
What it does not show
The outcome measured is developer click behaviour, which the authors call “an imperfect measure of analyzer quality” — fixes made in the editor or before review are undercounted. It does not measure downstream defects or incidents prevented. Single organisation with unusually centralised build and review tooling.
Caitlin Sadowski, Jeffrey van Gogh, Ciera Jaspan, Emma Söderberg, Collin Winter
Where the analysis lands decides whether it is used. A prior non-integrated tool “was used by only 35 developers in 2014 (and by 20 of those only once)”; moving analysis into the review UI with a one-click not-useful button and a hard false-positive bar drove sustained engagement and a measurable decline in flagged patterns. Successful analysers held a not-useful rate between 0 and 3%.
Tier III: Single-organisation engineering report, but with in-situ quantitative telemetry over 90 days rather than anecdote. No cross-organisation or controlled comparison.