Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark

By Nikita Nangia, Samuel R. Bowman. 2019

76 citations recorded by the source corpus.

The GLUE benchmark (Wang et al., 2019b) is a suite of language understanding tasks which has seen dramatic progress in the past year, with average performance moving from 70.0 at launch to 83.9, state of the art at the time of writing (May 24, 2019). Here, we measure human performance on the benchmark, in order to learn whether significant headroom remains for further progress. We provide a conservative estimate of human performance on the benchmark through crowdsourcing: Our annotators are non-experts who must learn each task from a brief set of instructions and 20 examples. In spite of limited training, these annotators robustly outperform the state of the art on six of the nine GLUE tasks and achieve an average score of 87.1. Given the fast pace of progress however, the headroom we observe is quite limited. To reproduce the data-poor setting that our annotators must learn in, we also train the BERT model (Devlin et al., 2019) in limited-data regimes, and conclude that low-resource sentence classification remains a challenge for modern neural network approaches to text understanding.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “…as of May 2019, except on tasks which we recast (WSC), resplit (CB), or achieve Human Performance Pilehvar and Camacho-Collados (2019), Khashabi et al. (2018), Nangia and Bowman (2019), and Zhang et al. (2018) respectively provide estimates for human performance on WiC, MultiRC, RTE, and ReCoRD.”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, neutral): “octoral researcher). The average accuracy was 76% for Mechanical Turk participants and 97% for expert annotators; further details are in Appendix F. Our Mechanical Turk results contrast with those of Nangia and Bowman (2019), who report an accuracy of 92% in the same population on examples from MNLI”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “Human performance For XNLI, PAWS-X, and XQuAD, we obtain human performance estimates from the English datasets they are derived from, MNLI, PAWS-X, and SQuAD respectively (Nangia & Bowman, 2019; Zhang et al., 2019; Rajpurkar et al., 2016).”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “The GLUE benchmark (Wang et al., 2019b; Nangia and Bowman, 2019), a compilation of NLU evaluation tasks, has seen performance on its leaderboard approach or exceed human performance on all nine of its tasks.”
Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (2019, background): “On the other hand, the state-of-the-art performance on common NLI datasets is already very close to human performance (Nikita and Bowman, 2019), suggesting that new datasets, such as the one presented here, are necessary to expose the models’ remaining limitations.”
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models (2021, methodology): “The annotation instructions for each task follows [37], and we provide at least two examples for each class to help annotators understand the tasks.”
OCNLI: Original Chinese Natural Language Inference (2020, methodology): “We use the score from undergraduate students as our human baseline as it is the “unbiased” score obtained using the same procedure as Nangia and Bowman (2019).”
Quantifying Gender Bias in Different Corpora (2020, background): “with humans and show that although GLUE tasks have improved recently, there is a gap between human and machine performance in Nangia and Bowman (2019) [8].”
Comparing Test Sets with Item Response Theory (2021, methodology): “For other dataset evaluations, in addition to providing a benchmark, the SuperGLUE paper also compares a set of candidate datasets using a ﬁxed pool of machine learning models and human annotators (Nangia and Bowman, 2019).”
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2023, background): “In comparison, Nangia and Bowman (2019) conservatively estimate human performance on XNLI to be 92 .”