The Effect of Natural Distribution Shift on Question Answering Models

By John Miller, Karl Krauth, B. Recht, Ludwig Schmidt. 2020

158 citations recorded by the source corpus.

We build four new test sets for the Stanford Question Answering Dataset (SQuAD) and evaluate the ability of question-answering systems to generalize to new data. Our first test set is from the original Wikipedia domain and measures the extent to which existing systems overfit the original test set. Despite several years of heavy test set re-use, we find no evidence of adaptive overfitting. The remaining three test sets are constructed from New York Times articles, Reddit posts, and Amazon product reviews and measure robustness to natural distribution shifts. Across a broad range of models, we observe average performance drops of 3.8, 14.0, and 17.4 F1 points, respectively. In contrast, a strong human baseline matches or exceeds the performance of SQuAD models on the original domain and exhibits little to no drop in new domains. Taken together, our results confirm the surprising resilience of the holdout method and emphasize the need to move towards evaluation metrics that incorporate robustness to natural distribution shifts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

In Natural Language Processing

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “While Hendrycks et al. (2020b) has reported that pre-training improves relative robustness on sentiment analysis, Miller et al. (2020)’s study of the robustness of question answering models under natural distribution shift finds, similar to Taori et al. (2020), little evidence of effective…”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “[58] Miller, J.”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “More recently, Taori et al. (2020) studied the robustness of image classification models, and Miller et al. (2020) investigated this for question-answering models.”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Others have studied model generalization to test sets drawn from different types of sources as the training set, e.g., Wikipedia, Reddit, news articles, travel reviews, and so on (Oren et al., 2019; Miller et al., 2020).”
Robust fine-tuning of zero-shot models (2021, background): “In contrast, accuracy on the reference distribution is often a reliable predictor for accuracy under distribution shift [67, 68, 92, 95, 104].”
Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches (2021, background): “…danger of adaptive overﬁtting due to test set reuse (Dwork et al., 2015), a recent line of work examines whether systems have overﬁt to particular test sets by taking existing systems and testing them on newly-constructed test sets (Recht et al., 2019; Yadav and Bottou, 2019; Miller et al., 2020).”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, methodology): “Similarly, models trained for the original SQuAD dataset performed poorly when evaluated on data collected from different source domains, such as Amazon crowd reviews and Reddit posts [97].”
The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning (2021, background): “The linear relationship holds across a wide range of models and is well-established for several robustness benchmarks, including image classiﬁcation on both synthetic and natural distribution shifts [32, 47, 33, 41], 2D object detection [36], and question-answer models in NLP [24].”
Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings (2022, background): “They often make incorrect predictions, especially when inputs tend to diverge from their training data distribution (Elsahar and Gallé, 2019; Miller et al., 2020; Koh et al., 2021).”
Utility Is in the Eye of the User: A Critique of NLP Leaderboard Design (2020, background): “…has proposed changes to prevent over-fitting via multiple resubmissions (Hardt and Blum, 2015; Hardt, 2017) while some have argued that this issue is overblown, given that question-answering systems submitted to the SQuAD leaderboard do not over-fit to the original test set (Miller et al., 2020).”
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, background): “Recent work has indicated that improvements on these high-quality benchmark datasets is not the result of overfitting, and is a reliable indicator of genuine progress beyond the benchmark data [77, 78, 79, 80].”