SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

By Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. Bowman. 2019

2,854 citations recorded by the source corpus.

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at this http URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “This has led to the development of a wide variety of pre-training objectives [Dai and Le, 2015; Ramachandran et al., 2016; Radford et al., 2018; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019b; Wang et al., 2019a; Song et al., 2019; Dong et al., 2019; Joshi et al., 2019].”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “GLUE The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019b) is a collection of 9 datasets for evaluating natural language understanding systems.”
Scaling Laws for Neural Language Models (2020, background): “…learning has recently seen rapid progress in language modeling, with state of the art models [RNSS18, DCLT18, YDY + 19, LOG + 19, RSR + 19] approaching human-level performance on many speciﬁc tasks [WPN + 19], including the composition of coherent multi-paragraph prompted text samples [RWC + 19].”
PaLM: Scaling Language Modeling with Pathways (2022, background): “…DROP (Dua et al., 2019), CoQA (Reddy et al., 2018), QuAC (Choi et al., 2018), SQuADv2 (Rajpurkar et al., 2018), RACE (Lai et al., 2017) • SuperGLUE (Wang et al., 2019a) • Natural Language Inference (NLI): Adversarial NLI (Nie et al., 2019) Table 4 includes the results for the PaLM 540B…”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “A single, pre-trained language model has been shown to achieve strong performance on various classiﬁcation tasks in the GLUE benchmarks [54, 55] after ﬁne-tuning [43, 8].”
How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “Discourag-ingly, even though GLUE reports such task-speciﬁc scores—and even includes diagnostic examples created by experts—these ﬁner-grain results have failed to gain the same traction as the headline GLUE benchmark.”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “…and approximately 7 years to surpass humans on ImageNet (Deng et al., 2009; Russakovsky et al., 2015; He et al., 2016), the GLUE benchmark did not last as long as we would have hoped after the advent of BERT (Devlin et al., 2018), and rapidly had to be extended into Super-GLUE (Wang et al., 2019).”
WinoGrande (2019, methodology): “The double dagger (‡) denotes that we could not reproduce the same number as in SuperGLUE leaderboard (Wang et al. 2019).”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, neutral): “…(Reddy et al., 2019), QuAC (Choi et al., 2018), SQuADv2 (Rajpurkar et al., 2018), RACEh (Lai et al., 2017), RACE-m (Lai et al., 2017) SuperGLUE: (Wang et al., 2019) BoolQ (Clark et al., 2019), CB (de Marneffe et al., 2019), COPA (Gordon et al., 2012), RTE (Dagan et al., 2006), WiC (Pilehvar &…”
Ethical and social risks of harm from Language Models (2021, background): “2.3.2 Disseminating false or misleading information Allcott et al. (2019); Krittanawong et al. (2020); Wang et al. (2019b) Branwen (2020); Dale (2021); Lacker (2020); Lin et al. (2021); Rae et al. (2021); Zhang et al. (2021b) 2.3.3 Causing material harm by disseminating misinformation e.g. in…”
Scaling Laws for Transfer (2021, methodology): “Given this analysis, it’s not surprising that few-shot scores on SuperGLUE [WPN + 19] were 10-15 points higher than zero-shot scores [BMR + 20].”
Measuring Massive Multitask Language Understanding (2020, methodology): “To address the shortcomings of GLUE, researchers designed the SuperGLUE benchmark with more difﬁcult tasks (Wang et al., 2019).”
AI and the Everything in the Whole Wide World Benchmark (2021, methodology): “Benchmarks like GLUE [Wang et al., 2019a] or ImageNet [Deng et al., 2009] are often elevated to become deﬁnitions of the essential common tasks to validate the performance of any given model.”
Training Verifiers to Solve Math Word Problems (2021, background): “In recent years, large language models have demonstrated impressive skills across many diverse tasks (Wang et al., 2019; Brown et al., 2020).”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “As further analysis, we train continuous prompts for each of the SuperGLUE (Wang et al., 2019a) tasks in accordance with the cluster splits from §2.2 such that when prompt-tuning on task T , no tasks in the same cluster as T were seen during instruction tuning.”