Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?

By Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, Jordan L. Boyd-Graber. 2021

136 citations recorded by the source corpus.

Leaderboards are widely used in NLP and push the field forward. While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items (examples) and subjects (NLP models). Rather than replace leaderboards, we advocate a re-imagining so that they better highlight if and where progress is made. Building on educational testing, we create a Bayesian leaderboard model where latent subject skill and latent item difficulty predict correct responses. Using this model, we analyze the ranking reliability of leaderboards. Afterwards, we show the model can guide what to annotate, identify annotation errors, detect overfitting, and identify informative examples. We conclude with recommendations for future benchmark tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Item response theory (IRT; Embretson & Reise, 2013) allows the difficulty of instances to be learned via parameters in a probabilistic model meant to explain model performance (Lalor et al., 2018; Rodriguez et al., 2021).”
Comparing Test Sets with Item Response Theory (2021, background): “In concurrent work, Rodriguez et al. (2021) study how IRT can be used for more nuanced leaderboard evaluations.”
Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy (2021, background): “Considering such shortcomings, Rodriguez et al. (2021) advocate improving evaluation by focusing on more important examples for ranking models; demographic properties could further refine more holistic evaluations.”
Evaluation Gaps in Machine Learning Practice (2022, background): “Furthermore, within each benchmark, items are weighted equally (thus focusing on the head of the data distribution), failing to capture inherent differences in difficulty across items, and hence providing poor measures of progress on task performance [141].”
ANLIzing the Adversarial Natural Language Inference Dataset (2020, background): “other modes of evaluation that explore model and data dynamics (Gardner et al., 2020; Swayamdipta et al., 2020; Rodriguez et al., 2021).”
Information-Theoretic Measures of Dataset Difficulty (2021, background): “Other work has treated instance difﬁculty as learned parameters, allowing estimates to vary depending on what the generative process and parameter distributions are assumed to be (Lalor et al., 2018; Rodriguez et al., 2021).”