Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
https://arxiv.org/pdf/1810.11953
Method
"We focus on several datasets and various perturbations to both covariates and label distributions with varying magnitudes and fractions of data affected."
Population
"Our main experiments were carried out on the MNIST (Ntr = 50000; Nval = 10000; Nte = 10000; D = 28 x 28 x 1; C = 10 classes)"
What it does not show
Image classification benchmarks with synthetic, researcher-injected shifts of known type and magnitude. Does not demonstrate performance on organic production drift, on tabular, text or time-series data, or in a streaming pipeline. Malignancy is judged by accuracy drop only, not by fairness or calibration shift.
Stephan Rabanser, Stephan Günnemann, Zachary C. Lipton
Two-sample testing using a pre-trained classifier for dimensionality reduction performed best; the popular domain-classifier approach performed badly below about 100 samples and only caught up with more. Applied to the canonical benchmark’s own train/test split, the detector found a real shift traced to how one digit was drawn between collection batches, at p = 2.7e-10 — though its own malignancy test judged it harmless to accuracy.
Tier I: Controlled experiment: known shifts injected at controlled magnitudes and fractions into held-out data, with multiple detection methods compared against that ground truth across sample sizes.