GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
By Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. Bowman. 2018
8,768 citations recorded by the source corpus.
Human ability to understand language is general, flexible, and robust. In contrast, most NLU models above the word level are designed for a specific task and struggle with out-of-domain data. If we aspire to develop models with understanding beyond the detection of superficial correspondences between inputs and outputs, then it is critical to develop a unified model that can execute a range of linguistic tasks across different domains. To facilitate research in this direction, we present the General Language Understanding Evaluation (GLUE, gluebenchmark.com): a benchmark of nine diverse NLU tasks, an auxiliary dataset for probing models for understanding of specific linguistic phenomena, and an online platform for evaluating and comparing models. For some benchmark tasks, training data is plentiful, but for others it is limited or does not match the genre of the test set. GLUE thus favors models that can represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks. While none of the datasets in GLUE were created from scratch for the benchmark, four of them feature privately-held test data, which is used to ensure that the benchmark is used fairly. We evaluate baselines that use ELMo (Peters et al., 2018), a powerful transfer learning technique, as well as state-of-the-art sentence representation models. The best models still achieve fairly low absolute scores. Analysis with our diagnostic dataset yields similarly weak performance over all phenomena tested, with some exceptions.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Graham Neubig
- Noah A. Smith
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Graham Neubig
- Jianfeng Gao
- Percy Liang
- Noah A. Smith
- Hannaneh Hajishirzi
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Samuel R. Bowman
- Luke S. Zettlemoyer
- Douwe Kiela
- Jianfeng Gao
- Hannaneh Hajishirzi
- Ashish Sabharwal
Citations in the corpus
- SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “As a result, models trained on such datasets with hu-1 man biases run the risk of over-estimating the actual performance on the underlying task, and are vulnerable to adversarial or out-of-domain examples (Wang et al., 2018; Glockner et al., 2018).”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “RTE and WNLI are two of the tasks where machine performance has historically lagged behind human performance, which is 93.6 and 95.9 respectively [Wang et al., 2018].”
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “GLUE The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019b) is a collection of 9 datasets for evaluating natural language understanding systems.”
- XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “The GLUE dataset [34] is a collection of 9 natural language understanding tasks.”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “A single, pre-trained language model has been shown to achieve strong performance on various classification tasks in the GLUE benchmarks [54, 55] after fine-tuning [43, 8].”
- How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “have obtained rapidly increasing scores on language understanding benchmarks such as GLUE (Wang et al., 2019b) or SuperGLUE (Wang et al.”
- Adversarial Training for Large Neural Language Models (2020, methodology): “To assess the impact of adversarial training on generalization, we use standard benchmarks such as GLUE (Wang et al., 2018) and SQuAD (v1.”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “We also evaluate the model by finetuning on tasks from GLUE (Wang et al., 2019a).”
- Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “Finally, we also collected annotations using the longer contexts present in the GLUE RTE training data, which came from the RTE5 dataset (Bentivogli et al., 2009).”
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “It matches the performance of RoBERTa (Liu et al., 2019) with comparable training resources on GLUE (Wang et al., 2018) and SQuAD (Rajpurkar et al., 2016), and achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks.”
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019, background): “General Language Understanding We assess the language understanding and generalization capabilities of DistilBERT on the General Language Understanding Evaluation (GLUE) benchmark [Wang et al., 2018], a collection of 9 datasets for evaluating natural language understanding systems.”
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “(2019), we evaluate our models on three popular benchmarks: The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018), two versions of the Stanford Question Answering Dataset (SQuAD; Rajpurkar et al.”
- TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “The GLUE datasets are described as follows: MNLI.”
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “We finetune the trained models on several downstream tasks including MNLI and QQP from the GLUE benchmark (Wang et al., 2019), SQuAD 1.1 and SQuAD 2.0 from the Stanford Question answering dataset (Rajpurkar et al., 2016; 2018), and the reading comprehension RACE dataset (Lai et al., 2017).”
- Defending Against Neural Fake News (2019, methodology): “11For each discriminator/generator pair, we search over p P t.9, .92, .94, .96, .98, 1.0u. 12Indeed, bidirectional approaches perform best on leaderboards like GLUE (Wang et al., 2018). discriminating between real and generated news articles suggests that neural fake news discrimination requires hav”
- Story Ending Prediction by Transferable BERT (2019, methodology): “STILTs [ Phang et al. , 2018 ] fine-tuned a GPT model on some intermediate tasks to get better performance on the GLUE [ Wang et al. , 2018 ] benchmark.”