The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions

By Xiang Zhou, Yixin Nie, Hao Hao Tan, Mohit Bansal. 2020

42 citations recorded by the source corpus.

We find that the performance of state-of-the-art models on Natural Language Inference (NLI) and Reading Comprehension (RC) analysis/stress sets can be highly unstable. This raises three questions: (1) How will the instability affect the reliability of the conclusions drawn based on these analysis sets? (2) Where does this instability come from? (3) How should we handle this instability and what are some potential solutions? For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve. We also observe lower-than-expected correlations between the analysis validation set and standard validation set, questioning the effectiveness of the current model-selection routine. Next, to answer the second question, we give both theoretical explanations and empirical evidence regarding the source of the instability, demonstrating that the instability mainly comes from high inter-example correlations within analysis sets. Finally, for the third question, we discuss an initial attempt to mitigate the instability and suggest guidelines for future work such as reporting the decomposed variance for more interpretable results and fair comparison across models. Our code is publicly available at: this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, neutral): “Within each functionality, we aim to use diverse vocabulary and syntax to reduce similarity between test cases, which Zhou et al. (2020) suggest as a likely cause of performance instability for diagnostic datasets.”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “…on these stress tests: first, the magnitude of the variation is large; second, the variation is sensitive to the fine-tuning seed, replicating Zhou et al. (2020); third, the variation is also sensitive to the pretraining seed; fourth, the variation is difficult to predict based on…”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “Yet, challenge sets may suffer from performance instability (Liu et al., 2019a; Rozen et al., 2019; Zhou et al., 2020) and often lack sufficient statistical power (Card et al.”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Pathology datasets like Camelyon17-wilds tend to comprise a small number of slides (though a large number of patches extracted from these slides), and there is a lot of correlation across patches from the same slides and hospitals, which can exacerbate this instability (Zhou et al., 2020).”
ANLIzing the Adversarial Natural Language Inference Dataset (2020, background): “…datasets (McCoy et al., 2019; Naik et al., 2018; Nie et al., 2019; Yanaka et al., 2019; Warstadt et al., 2019; Jeretic et al., 2020; Warstadt et al., 2020); see Zhou et al. (2020) for a critical exami- 2Assuming that models trained on later rounds don’t suffer from catastrophic forgetting. nation.”
Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structures (2020, background): “As reported by McCoy et al. (2019a) and Zhou et al. (2020), the performance on the adversarial evaluation sets can highly vary given different values of hyper-parameters.”
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “Zhou et al. (2020) suggest similarity between test cases as a cause of performance instability in diagnostic datasets.”
Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization (2021, methodology): “More recently, empirical studies have shown that in some settings, models with similar in-distribution performance can indeed have different out-of-distribution performance (Mc-Coy et al., 2019; Zhou et al., 2020; D’Amour et al., 2020).”
The MultiBERTs: BERT Reproductions for Robustness Analysis (2021, background): “Understanding this difference is critical if we are to generate reusable insights about deep learning for NLP, and improve the state-of-the-art going forward (Zhou et al., 2020; Dodge et al., 2020).”