A Dissection of the Test-Driven Development Process: Does It Really Matter to Test-First or to Test-Last?
https://arxiv.org/pdf/1611.05994
Method
"This study is neither a controlled experiment nor a quasi-experiment. No experimental groups (control and treatment groups) can be created by manipulating the factors... Ordinarily, observational studies reveal associations: their power to reveal causality is limited."
Population
"We analyzed 82 data points collected from 39 professionals, each capturing the process while performing a specific development task."
What it does not show
Not a controlled experiment by the authors’ own statement. Professionals were being taught the technique in a workshop, which may not reflect steady-state practice. Quality measured only as functional correctness. Like the 2005 experiment, it addresses test-first ordering in feature work, not bug-reproduction-first.
Davide Fucci, Hakan Erdogmus, Burak Turhan, Markku Oivo, Natalia Juristo
“Sequencing, the order in which test and production code are written, had no influence” on quality or productivity. What did predict outcomes was granularity and uniformity of cycles — suggesting the benefits attributed to test-first come from working in fine, steady steps rather than from the ordering itself.
Tier II: The authors explicitly state this is neither a controlled experiment nor a quasi-experiment: a regression-based observational study of professionals with no random assignment.