GLGE: A New General Language Generation Evaluation Benchmark

By Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong and 8 more. 2020

75 citations recorded by the source corpus.

Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP). These benchmarks mostly focus on a range of Natural Language Understanding (NLU) tasks, without considering the Natural Language Generation (NLG) models. In this paper, we present the General Language Generation Evaluation (GLGE), a new multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. For each task, we continue to design three subtasks in terms of task difficulty (GLGE-Easy, GLGE-Medium, and GLGE-Hard). This introduces 24 subtasks to comprehensively compare model performance. To encourage research on pretraining and transfer learning on NLG models, we make GLGE publicly available and build a leaderboard with strong baselines including MASS, BART, and ProphetNet\footnote{The source code and dataset will be publicly available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “This shortfall can be seen in benchmarks like DecaNLP (McCann et al., 2018) and GLGE (Liu et al., 2020a) which include NLG tasks but focus only on a single metric and, as a result, may mischaracterize a system’s performance.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Inspired by the success of GLUE and SuperGLUE benchmarks in the general English domain, benchmarks like GENIE [262], GEM [263], GLGE [264] have been introduced to evaluate NLG models in the general English domain.”
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021, background): “Several emerging NLG benchmarks (Gehrmann et al., 2021; Liu et al., 2021) collected existing metrics for various tasks, whereas we aim at developing new uniﬁed metrics with stronger performance.”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “Liu et al. [143] introduced General Language Generation Evaluation (GLGE) , a multi-task text generation benchmark containing eight English language generation tasks.”