Evaluation of Text Generation: A Survey

By Asli Celikyilmaz, Elizabeth Clark, Jianfeng Gao. 2020

476 citations recorded by the source corpus.

The paper surveys evaluation methods of natural language generation (NLG) systems that have been developed in the last few years. We group NLG evaluation methods into three categories: (1) human-centric evaluation metrics, (2) automatic metrics that require no training, and (3) machine-learned metrics. For each category, we discuss the progress that has been made and the challenges still being faced, with a focus on the evaluation of recently proposed NLG tasks and neural NLG models. We then present two case studies of automatic text summarization and long text generation, and conclude the paper by proposing future research directions.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Learning Compact Metrics for MT (2021, neutral): “This has led to a surge of interest in learned metrics that cast evaluation as a regression problem and leverage pre-trained multilin-∗ gual models to capture the semantic similarity be-tween references and generated text (Celikyilmaz et al., 2020).”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “In recent work, Howcroft et al. (2020) investigated NLG papers from the last 2For a more complete description of recent developments in NLG evaluation, we refer to the survey by Celikyilmaz et al. (2020). twenty years and the evaluation methodologies differ drastically across papers.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “, 2022), which are well known to pose unique challenges for automatic evaluation (Lowe et al., 2017; Howcroft et al., 2020; Celikyilmaz et al., 2021).”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “We now present a comprehensive evaluation of our LongForm models compared to the baselines on LongForm-C test, recipe generation, long form question answering, short story generation, NLU, and ablation of LongForm-C subsets.”
Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions (2020, background): “…(Celikyilmaz et al., 2020; Gatt and Krahmer, 2018), but come with their own issues, such as cost and time involved, the need for domain expertise (Celikyilmaz et al., 2020), and the fact that the experimental setup has a substantial impact on the reliability of human quality judgements (Novikova…”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “Ultimately, all automatic metrics have their limitations, and human evaluation is necessary (Celikyilmaz et al., 2020).”
Q^{2}: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021, background): “Factual inconsistency is often overlooked by standard evaluation methods for text generation (Celikyilmaz et al., 2020), and dialogue systems are no exception.”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “Recent efforts by NLP researchers have drawn attention to the issue of factual errors and hallucinations in the output of neural summarization models (Cao et al., 2018; Massarelli et al., 2019; Zhao et al., 2020; Falke et al., 2019b; Goodrich et al., 2019; Celikyilmaz et al., 2020).”
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022, background): “Then, we describe potential use cases of CoAuthor in HCI and NLP research, exemplifying the potential of CoAuthor serving as a boundary object.”
Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task (2020, background): “This has led to a surge of interest for more flexible metrics that use machine learning to capture semantic-level information (Celikyilmaz et al., 2020).”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, background): “A well-developed metric can efficiently indicate the model performance (Celikyilmaz et al., 2020) and guide the text generation (Unanue et al.”
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, methodology): “Finally, since automatic metrics for text generation are limited (Celikyilmaz et al., 2020), we additionally conduct a human evaluation of DIPPER’s”
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation (2022, unknown): “Following the taxonomies reported by Celikyilmaz et al. (2020) and Sai et al. (2020), the metrics considered can be loosely grouped in: • Text overlap metrics.”
Dialect Diversity in Text Summarization on Twitter (2020, background): “Recent work has proposed summary generation methods that are factually consistent [11] and extensions of our post-processing framework for such methods can explored as part of future work.”
RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “human evaluation remains critical for open-ended generation (Celikyilmaz et al., 2020; Gehrmann et al., 2022).”
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022, unknown): “Consistency aims to evaluate whether the generated text is consistent to the content preﬁx (Celikyilmaz et al., 2020; Rashkin et al., 2020).”
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers (2021, background): “They capture task-specific properties; see the surveys [8, 48].”
"I'm sorry to hear that": finding bias in language models with a holistic descriptor dataset (2022, background): “…(Perez et al., 2022; Xu et al., 2021b; Kirk et al., 2021a; Sheng et al., 2021b; Nozza et al., 2021; Renduchintala et al., 2021; Dinan et al., 2020a,b), which are well known to pose unique challenges for automatic evaluation (Lowe et al., 2017; Howcroft et al., 2020; Celikyilmaz et al., 2021).”