We Need To Talk About Random Splits

By Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, Katja Filippova. 2020

113 citations recorded by the source corpus.

(CITATION) argued for using random splits rather than standard splits in NLP experiments. We argue that random splits, like standard splits, lead to overly optimistic performance estimates. We can also split data in biased or adversarial ways, e.g., training on short sentences and evaluating on long ones. Biased sampling has been used in domain adaptation to simulate real-world drift; this is known as the covariate shift assumption. In NLP, however, even worst-case splits, maximizing bias, often under-estimate the error observed on new samples of in-domain data, i.e., the data that models should minimally generalize to at test time. This invalidates the covariate shift assumption. Instead of using multiple random splits, future benchmarks should ideally include multiple, independent test sets instead; if infeasible, we argue that multiple biased splits leads to more realistic performance estimates than multiple random splits.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “For example, the the Language Interpretability Tool (Tenney et al., 2020) can automatically evaluate on subpopulations, and Explainaboard (Liu et al., 2021), although with limited support for NLG, helps identifying challenging inputs for a model.”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “…(Rudinger et al., 2018), popularity in named entity disambiguation (Chen et al., 2021), similarity between training and test instances in information extraction and sentiment analysis datasets (Elangovan et al., 2021), and effects of how data is split (Gorman & Bedrick, 2019; Søgaard et al., 2021).”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, methodology): “…evaluating on future text) for NLP tasks like machine translation (Levenberg et al., 2010), sentiment analysis (Lukes and Søgaard, 2018), named entity recognition (Fromreide et al., 2014; Rijhwani and Preotiuc-Pietro, 2020), and others (Dunn et al., 2017; Bjerva et al., 2020; Søgaard et al., 2021).”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, methodology): “Validation & testing The need for multiple test sets to capture performance on a variety of down-stream conditions has already been recommended by recent work (Søgaard et al., 2021; Lazaridou et al., 2021).”
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets (2021, background): “10 These ﬁndings are in line with those by [58], and by [18] who ﬁnd that newly collected data commonly leads to worse performance than even adversarial examples.”
Pitfalls of Static Language Modelling (2021, background): “Second, as new and ever-larger datasets are presently compiled using web crawls (Radford et al., 2019; Gao et al., 2021), it has never been more timely to rethink how our splits are constructed (Søgaard et al., 2020).”
Randomness In Neural Network Training: Characterizing The Impact of Tooling (2021, methodology): “, 2019) and test set construction (Søgaard et al., 2021; Lazaridou et al., 2021; Melis et al., 2018).”