The Benchmark Lottery

By Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, Oriol Vinyals. 2021

147 citations recorded by the source corpus.

The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of"a benchmark lottery"that describes the overall fragility of the ML benchmarking process. The benchmark lottery postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior. On multiple benchmark setups that are prevalent in the ML community, we show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks, highlighting the fragility of the current paradigms and potential fallacious interpretation derived from benchmarking ML methods. Given that every benchmark makes a statement about what it perceives to be important, we argue that this might lead to biased progress in the community. We discuss the implications of the observed phenomena and provide recommendations on mitigating them using multiple machine learning domains and communities as use cases, including natural language processing, computer vision, information retrieval, recommender systems, and reinforcement learning.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “Verifying whether the findings generalize outside GLUE and SuperGLUE is important since we do not want to fall prey to the benchmark lottery problem (Dehghani et al., 2021).”
UL2: Unifying Language Learning Paradigms (2022, background): “It is clear from these results that encoder-decoder models should be preferred over decoder-only models if and only if there is no concern about storage, i.e., parameter counts are generally less important than actual throughput (see (Dehghani et al., 2021a) for a detailed discussion).”
The Efficiency Misnomer (2021, background): “The most common knobs for scaling up models are changing depth (number of layers) and width (hidden dimension) of models (Tay et al., 2021b).”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, background): “Furthermore, this process ignores properties that do not match the leaderboard metric (Ethayarajh and Ju-rafsky, 2020; Bowman and Dahl, 2021; Dehghani et al., 2021).”
Unifying Language Learning Paradigms (2022, background): “We also highlight the fact that the overall gain is normalized , so this becomes harder to exploit or be susceptible to benchmark lottery effects (Dehghani et al., 2021b).”
When Combating Hype, Proceed with Caution (2021, background): “Welty et al. (2019), Linzen (2020), Ribeiro et al. (2020), Raji et al. (2021), Bowman and Dahl (2021), and Dehghani et al. (2021), among many others, discuss the challenges involved in designing evaluations that yield trustworthy and accurate depictions of the capabilities of ML models.”
Harms from Increasingly Agentic Algorithmic Systems (2023, background): “Dehghani et al. [2021] provide evidence that factors other than “fundamental algorithmic superiority” may lead to the perception that a particular method is superior.”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, methodology): “This method tries to evaluate the performance of LLMs on various benchmark datasets [108, 109] and demonstrate their efficiency.”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, background): “[24] Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals.”
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024, background): “However, the current generation of leaderboards still do not satisfy many of the requirements of researchers and practitioners looking to build on LLMs (Ethayarajh and Juraf-sky, 2021; Dehghani et al., 2021).”
SCENIC: A JAX Library for Computer Vision Research and Beyond (2021, methodology): “On top of that, SCENIC has been used in numerous Google projects and research papers such as ViViT [6], OmniNet [28], TokenLearner [25], MBT [21], PolyViT [19], MTV [31], studies on scaling behaviour [5, 10, 29] and efficiency [9] of various models among others [15–18, 20].”
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2022, background): “…Transformers, this 12342 1.1e+122.2e+12 4.4e+12 8.8e+12 1.8e+13 3.5e+13 7.0e+131.4e+14 architectural choice significantly warps the scaling behaviour not only with respect to performance but also amongst compute metrics such as FLOPs, speed and number of parameters (Dehghani et al., 2021a).”