We Need to Talk about Standard Splits

By Kyle Gorman, Steven Bedrick. 2019

130 citations recorded by the source corpus.

It is standard practice in speech & language technology to rank systems according to their performance on a test set held out for evaluation. However, few researchers apply statistical tests to determine whether differences in performance are likely to arise by chance, and few examine the stability of system ranking across multiple training-testing splits. We conduct replication and reproduction experiments with nine part-of-speech taggers published between 2000 and 2018, each of which claimed state-of-the-art performance on a widely-used “standard split”. While we replicate results on the standard split, we fail to reliably reproduce some rankings when we repeat this analysis with randomly generated training-testing splits. We argue that randomly generated splits should be used in system evaluation.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “As a way to make i.i.d. schemes more robust, Gorman and Bedrick (2019) propose using multiple random splits similar to cross-validation as they find that results on multiple NLU tasks change when the splitting process is changed.”
Show Your Work: Improved Reporting of Experimental Results (2019, background): “Reproducibility issues relating to standard data splits (Schwartz et al., 2011; Gorman and Bedrick, 2019; Recht et al., 2019a,b) have surfaced in a number of areas.”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “…(Rudinger et al., 2018), popularity in named entity disambiguation (Chen et al., 2021), similarity between training and test instances in information extraction and sentiment analysis datasets (Elangovan et al., 2021), and effects of how data is split (Gorman & Bedrick, 2019; Søgaard et al., 2021).”
SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020) (2020, background): “The data is not divided into distinct train and development splits, but rather, system builders are encouraged to perform cross-validation, in an attempt to reduce the arte-facts that standard splits can introduce (Gorman and Bedrick, 2019).”
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets (2021, background): “Gorman et al. [17] point out that it can lead to incorrect system rankings and thus advocate training on multiple random splits.”
Leakage and the reproducibility crisis in machine-learning-based science (2023, background): “However, one undesirable outcome that has been observed in communities that have adopted the common task framework is a singular focus on optimizing a particular accuracy metric to the exclusion of other scientific and normatively desirable properties of models.(67,85,99) [D5] Lack of computational ”
We Need To Talk About Random Splits (2020, methodology): “In reality, what Gorman and Bedrick (2019) call the true difference in system performance, i.e., δ =M(Gtest , S1)−M(Gtest , S2), is the system difference on data that users would expect the systems to work well on (see §2 for practical examples) – and not just on the corpus that we happen to have…”
Probing Toxic Content in Large Pre-Trained Language Models (2021, background): “Other work investigate the effect of data splits (Gorman and Bedrick, 2019) and mitigation strategies (Dixon et al., 2018; Sun et al., 2019).”
Implicit Discourse Relation Classification: We Need to Talk about Evaluation (2020, background): “Gorman and Bedrick (2019) have proposed validation over randomized splits using signiﬁcance testing with multiple-comparisons correction.”