Monitoring Data Quality at Scale with Statistical Modeling
https://www.uber.com/blog/monitoring-data-quality-at-scale/
Method
"We use the prediction interval to validate whether the current one-step ahead value adheres to historical patterns."
Population
"Hosting tens of thousands of tables, it is not possible for us to manually assess the quality of each piece of back-end data in our pipelines."
What it does not show
No quantified false-positive rate, precision, recall, or count of real incidents caught versus missed — the rarer-alerts claim is qualitative. Does not describe how thresholds are revalidated as pipelines change.
Santhosh Shanmugam, Andrea Pasqua, Ritesh Agrawal, Ye Henry Li
Dimensionality reduction over multi-metric table health signals combined with exponential-smoothing forecasting, flagging one-step-ahead deviations across tens of thousands of tables where manual review is infeasible. Table-level alerts under this scheme are described as much rarer than per-metric alerts, framed explicitly as reducing alert fatigue.
Tier III: Single-organisation engineering report with scale figures but no quantified detection rate, precision or comparison group.