The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

By Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D. Dhole and 46 more. 2021

329 citations recorded by the source corpus.

We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “, tasks from GEM (Gehrmann et al., 2021), and TyDiQA (Clark et al., 2020).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “These findings has also been corroborated for XSum (Gehrmann et al., 2021; Mille et al., 2021).”
UL2: Unifying Language Learning Paradigms (2022, methodology): “For WebNLG, E2E and CommonGen, we use the versions from the GEM benchmark (Gehrmann et al., 2021).”
Measuring Attribution in Natural Language Generation Models (2021, background): “General-purpose benchmarking across these tasks have gained traction (Gehrmann et al., 2021), but there has not been a standardized treatment of the attribution problem.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “, 2020) from GEM generation benchmark (Gehrmann et al., 2021).”
Datasets: A Community Library for Natural Language Processing (2021, methodology): “Datasets was used to support the ﬁrst GEM (Generation, Evaluation, and Metrics) work-shop (Gehrmann et al., 2021).”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, neutral): “existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources (2022, background): “…been recently produced to aid NLP researchers in documenting their own datasets (Gao et al., 2020; Biderman et al., 1 http://commoncrawl.org/ 2022; Gehrmann et al., 2021; Wang et al., 2021) and even to retrospectively document and analyze datasets that were developed and released by others…”
ExplainaBoard: An Explainable Leaderboard for NLP (2021, background): “…et al., 2018, 2019), XTREME and XGLUE (Hu et al., 2020; Liang et al., 2020) for multilingual understanding, the WMT shared tasks (Barrault et al., 2020) for machine translation, and GEM and GENIE for natural language generation (Gehrmann et al., 2021; Khashabi et al., 2021), among many others.”
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022, background): “Then, we describe potential use cases of CoAuthor in HCI and NLP research, exemplifying the potential of CoAuthor serving as a boundary object.”