Utility Is in the Eye of the User: A Critique of NLP Leaderboard Design

By Kawin Ethayarajh, Dan Jurafsky. 2020

62 citations recorded by the source corpus.

Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models. While this leaderboard paradigm has been remarkably successful, a historical focus on performance-based evaluation has been at the expense of other qualities that the NLP community values in models, such as compactness, fairness, and energy efficiency. In this opinion paper, we study the divergence between what is incentivized by leaderboards and what is useful in practice through the lens of microeconomic theory. We frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. With this framing, we formalize how leaderboards -- in their current form -- can be poor proxies for the NLP community at large. For example, a highly inefficient model would provide less utility to practitioners but not to a leaderboard, since it is a cost that only the former must bear. To allow practitioners to better estimate a model's utility to them, we advocate for more transparency on leaderboards, such as the reporting of statistics that are of practical concern (e.g., model size, energy efficiency, and inference latency).

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, background): “Other dimensions include energy consumption (both for development and testing) [Strubell et al., 2019, Henderson et al., 2020, Schwartz et al., 2020, Ethayarajh and Jurafsky, 2020]; memory and compute requirements, which may be more or less constrained depending on the deployment context…”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “However, they also pose a risk that progress is reduced to the single number shown in a benchmark’s leaderboard and thus may encourage blindly optimizing it without regard to other considerations like model size or fairness (Ethayarajh and Jurafsky, 2020).”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “This will in turn enable dynamic leaderboards, for example based on utility (Ethayarajh and Jurafsky, 2020).”
Evaluation Gaps in Machine Learning Practice (2022, background): “Resist the temptation to reduce a model’s utility to a single scalar value, either for stack ranking [51] or to simplify the cognitive load on decision makers.”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “Within the NLP community, much of such recent discussion has focused on the nature of leaderboards and the practice of benchmarking (Ethayarajh and Jurafsky, 2020; Ma et al., 2021).”
What Can We Do to Improve Peer Review in NLP? (2020, background): “, 2020; Crane, 2018), no incentives for producing robust systems (Ethayarajh and Jurafsky, 2020), flawed benchmarks (Jia and Liang, 2017; McCoy et al.”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, background): “XTREME-R aims to move away from a single aggregate metric summarizing a model’s performance and towards a more nuanced evaluation and comparison of multilingual models (Ethayarajh and Jurafsky, 2020; Linzen, 2020).”