Adversarial exposure · Knowledge distribution
One reviewer is not enough for security-relevant code
A single manual reviewer finds a small minority of known vulnerabilities — thirty reviewers on the same codebase averaged 2.33 of the confirmed set, about a fifth found none at all, and none found them all.
Do this firstA review with no discussion is not a review
The experiment behind this is unusually clean: one codebase, a known set of planted vulnerabilities, thirty independently hired reviewers, a fixed time budget, and a count per reviewer.
The distribution is what makes the case. It is not that reviewers are bad — it is that any individual review is a sample of a large space, and different reviewers sample different parts of it. Only 17% spotted the missing cross-site-request-forgery protection. If your process assumes the reviewer who happened to be free will find the class of defect that matters, it is relying on a coin flip that the data says lands badly.
Independence is the active ingredient, and it is worth protecting: two people who review together converge, which recovers less of the space than two who review separately.
The decoy
A senior reviewer. Experience did not predict accuracy in the measurement: more experienced reviewers were not reliably better at finding the planted vulnerabilities or at avoiding false positives.
Evidence
- An Empirical Study on the Effectiveness of Security Code Review — INo reviewer found all confirmed vulnerabilities. The average found was 2.33 with a standard deviation of 1.67, about 20% found none at all, and only 17% found the missing cross-site request forgery protection. False-positive rates were bimodal, and more experience did not reliably mean more accurate or effective.
Last reviewed 2026-08-19.