Comparing Test Sets with Item Response Theory

By Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Sam Bowman, Samuel R. Bowman. 2021

64 citations recorded by the source corpus.

Recent years have seen numerous NLP datasets introduced to evaluate the performance of fine-tuned models on natural language understanding tasks. Recent results from large pretrained models, though, show that many of these datasets are largely saturated and unlikely to be able to detect further progress. What kind of datasets are still effective at discriminating among strong models, and what kind of datasets should we expect to be able to detect future improvements? To measure this uniformly across datasets, we draw on Item Response Theory and evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models, while SNLI, MNLI, and CommitmentBank seem to be saturated for current strong models. We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “Subsequent work highlights the need for further data collection efforts focusing on improving writing-based annotation processes (Vania et al., 2021), which we aim to investigate in this work.”
The Benchmark Lottery (2021, background): “…2021, Bouthillier et al., 2021, Recht et al., 2019, Lin et al., 2021, Balduzzi et al., 2018, Lipton and Steinhardt, 2018, Bouthillier et al., 2019, Vania et al., 2020, Mishra and Arunkumar, 2021, Marie et al., 2021, Dodge et al., 2019] are of utmost importance to correct empirical evaluation of…”
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021, methodology): “Following Vania et al. (2021), we use 90 Transformer-based models ﬁne-tuned on RACE, with varying ability levels, and use their predictions on our four datasets as responses.”