Detecting Data Errors: Where are we and what needs to be done?
https://www.vldb.org/pvldb/vol9/p993-abedjan.pdf
Method
"We have systematically run all the tools on all data sets and in this paper we report results (precision and recall) based on the ground truth (Section 4)."
Population
"We also obtained full or partial ground truth for each data set, so we can judge the performance and capabilities of available error detection tools."
What it does not show
A controlled benchmark on static datasets, not a live monitoring deployment, so it does not measure pipeline outcomes. It does not measure on-call burden directly, though its precision numbers quantify the mechanism that produces it. Says nothing about how hand-written expectation suites drift as distributions change.
Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F. Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, Nan Tang
Average precision across tools was about 47% and average recall about 36%, with no tool dominant. Taking the union of all tools raised recall to 85-99% and severely degraded precision — 99% recall at 39% precision on one dataset. A benefit-based ordering recovered 28% average precision for 3.5% recall loss. Some errors, such as a misspelled name or a wrong-but-plausible address, were structurally undetectable by any tool tested.
Tier I: Systematic comparison of eight error-detection tools across five real datasets against manually curated ground truth, with precision and recall computed per tool and per combination strategy. A genuine comparison-group design.