Dataset Shift in Machine Learning

By Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, Neil D. Lawrence. 2009

2,271 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Evaluation & Benchmarks

In Signal Processing

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “Such train-test mismatches occur because evaluation datasets rarely characterize the entire distribution (Torralba and Efros, 2011), and the test distribution typically drifts over time (Quionero-Candela et al., 2009).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Time Travel Dataset shifts are when the joint distribution of inputs and outputs differs between the development of a model and its deployment (or in our case test its test setup) (Quiñonero-Candela et al., 2009).”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “Domain adaptation aims to address the mismatch in data distribution between training and test domains (Daume III & Marcu, 2006; Quionero-Candela et al., 2009; Koh et al., 2021).”
On Robustness and Transferability of Convolutional Neural Networks (2020, background): “It is thus imperative to understand the impact dataset shifts [50] have on the performance of these models.”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “Reliable classiﬁcation under distribution shift is still out of reach for current machine learning [65, 68, 91].”
Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects (2018, neutral): “, not from the training distribution [42].”
Predicting Deep Zero-Shot Convolutional Neural Networks Using Textual Descriptions (2015, background): “[23, 24] address the problem mentioned above where access to both source and target data are available at training time.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “This problem is called distributional shift or dataset shift (Candela et al., 2009).”
Advances and Open Problems in Federated Learning (2019, background): “This taxonomy has a close mapping to notions of dataset shift [304, 327], which studies differences between the training distribution and testing distribution; here, we consider differences in the data distribution on each client.”
Learning explanations that are hard to vary (2020, background): “Covariate Shift [Sugiyama et al., 2007, Quionero-Candela et al., 2009, Sugiyama and Kawanabe, 2012] formalizes the case in which the distribution of the covariates at test time differs from that observed at training.”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “, 2017) under covariate shifts (Shimodaira, 2000; Quionero-Candela et al., 2009; Ben-David et al., 2010), fairer downstream predictions (Locatello et al.”
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms (2019, background): “Moreover, to make the case stronger, we will consider a setting called covariate shift (Rojas-Carulla et al., 2018; Quionero-Candela et al., 2009), where we assume that the change (again, unknown to the learner) between the training and transfer distributions occurs after an intervention on the caus”
Counterfactual Normalization: Proactively Addressing Dataset Shift and Improving Reliability Using Causal Mechanisms (2018, methodology): “aset shift which results in poor generalization. Most existing methods for addressing dataset shift are reactive: they use unlabeled data from the target distribution during the learning process (see Quionero-Candela et al. (2009) for an overview). However, when the differences in environments are u”
Conditional variance penalties and domain shift robustness (2017, background): “Issues can arise when the learned representations rely on dependencies that vanish in test distributions, see for example Quionero-Candela et al. (2009); Torralba and Efros (2011); Csurka (2017) and references therein.”
Domain Generalization via Invariant Feature Representation (2013, background): “In this case, the data exhibits covariate (or dataset) shift (Widmer and Kurat, 1996; Quionero-Candela et al., 2009; Bickel et al., 2009): although the marginal distributions PX on cell attributes vary due to biological or technical variations, the functional relationship P(Y |X) across different do”
Predictive Uncertainty Estimation via Prior Networks (2018, background): “Distributional uncertainty arises due to mismatch between the training and test distributions (also called dataset shift [24]) - a situation which often arises for real world problems.”