G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation

By Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, Daniel S. Weld. 2021

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “…to work toward reusability and replicability in human evaluations, for example by filling in human evaluation datasheets (Shimorina and Belz, 2021; Belz et al., 2021) and by contributing and using projects that standardize parts of the process (e.g., Khashabi et al., 2021; Gehrmann et al., 2021).”
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, background): “, 2020) and deployment (Khashabi et al., 2021; Gehrmann et al., 2021), will support future development of evaluator training procedures and the comparison of human evaluation results in future NLG evaluation work.”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, methodology): “Admitting limitations of automatic evaluations, we hope future work will provide an easy-to-reproduce human evaluation for the tasks studied here, based on the recent proposals for streamlining human evaluation of text generation models (Khashabi et al., 2021).”
UL2: Unifying Language Learning Paradigms (2022, methodology): “We evaluate on a variety of text generation tasks using human evaluation, via the GENIE leaderboard (Khashabi et al., 2021).”
Evaluation of Text Generation: A Survey (2020, methodology): “For example, GENIE (Khashabi et al., 2021) and GEM (Gehrmann et al.”
ExplainaBoard: An Explainable Leaderboard for NLP (2021, background): “…et al., 2018, 2019), XTREME and XGLUE (Hu et al., 2020; Liang et al., 2020) for multilingual understanding, the WMT shared tasks (Barrault et al., 2020) for machine translation, and GEM and GENIE for natural language generation (Gehrmann et al., 2021; Khashabi et al., 2021), among many others.”
Control Prefixes for Parameter-Efficient Text Generation (2021, methodology): “Therefore, we opted to submit our CONTROL PREFIXES model outputs to an externally run evaluation framework, GENIE (Khashabi et al., 2021), which provides an unbiased attestation of performance.”
SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization (2021, background): “Several recent works have realized the need for evolving benchmarks and evaluations (Goel et al., 2021; Gehrmann et al., 2021; Khashabi et al., 2021).”
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2021, background): “The human-in-the-loop G ENIE leaderboard (Khashabi et al., 2021) centralizes crowdsourced evaluations for generation tasks.”
Scarecrow: A Framework for Scrutinizing Machine Text (2021, methodology): “Recent work even automates this approach, running a human evaluation alongside automatic metrics on leaderboard submissions (Khashabi et al., 2021).”
News Summarization and Evaluation in the Era of GPT-3 (2022, unknown): “improvements on XSum on the GENIE leaderboard (Khashabi et al., 2021).”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, methodology): “See Section 5.1 of Khashabi et al., 2021).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Inspired by the success of GLUE and SuperGLUE benchmarks in the general English domain, benchmarks like GENIE [262], GEM [263], GLGE [264] have been introduced to evaluate NLG models in the general English domain.”
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge (2021, methodology): “We propose two ways to score answers to ARC-DA: The first is human scoring via GENIE6, a human-in-the-loop leaderboard framework that scores answers using an automated crowdsourced pipeline (Khashabi et al., 2021).”
Unifying Language Learning Paradigms (2022, methodology): “We evaluate on a variety of text generation tasks using human evaluation, via the GENIE leaderboard (Khashabi et al., 2021).”
A Survey of Knowledge-enhanced Text Generation (2020, methodology): “, GLGE [77], GEM [40], KilT [99], GENIE [60], and determined 9 benchmark datasets for evaluating knowledge-enhanced NLG methods.”