Comparing Automatic and Human Evaluation of NLG Systems

By Anya Belz, Ehud Reiter. 2006

274 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Learning Compact Metrics for MT (2021, neutral): “Since these metrics focus solely on surface-level aspects of the generated text, they correlate poorly with human evaluation, especially when models are producing high-quality text (Belz and Reiter, 2006; Callison-Burch et al., 2006; Ma et al., 2019; Mathur et al., 2020a).”
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, background): “We join past work (Belz and Reiter, 2006; van der Lee et al., 2021) in recommending a move towards evaluation settings where evaluators are better motivated to carefully consider the content and usefulness of generated text.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “, 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), and CIDEr (Vedantam et al.”
HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, methodology): “HTLM prompting of classi-ﬁcation datasets outperforms the most comparable (in terms of number of parameters) GPT-3 Medium sized model on the majority of tasks, while approaching—and on RTE outperforming— the GPT-3 Large model which consists of roughly double the amount of parameters as HTLM .”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, methodology): “For every tasks, we report the ofﬁcial evaluation metrics, including BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), TER (Snover et al., 2005), MoverScore (Zhao et al., 2019), BERTScore (Zhang et al.,…”
Why We Need New Evaluation Metrics for NLG (2017, background): “This is rarely the case, as shown by various studies in NLG (Stent et al., 2005; Belz and Reiter, 2006; Reiter and Belz, 2009), as well as in related fields, such as dialogue systems (Liu et al.”
Neural Text Generation from Structured Data with Application to the Biography Domain (2016, methodology): “Generation quality is assessed automatically with BLEU-4, ROUGE-4 (F-measure) and NIST-4 3 (Belz and Reiter, 2006).”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, background): “Over the years several studies have evaluated automatic metrics for measuring text generation performance (Callison-Burch et al., 2006; Stent et al., 2005; Belz and Reiter, 2006; Reiter, 2018; Liu et al., 2016; Kilickaya et al., 2017; Gatt and Krahmer, 2018).”
A Structured Review of the Validity of BLEU (2018, background): “For example, I excluded Belz and Reiter (2006), since Reiter and Belz (2009) presents an improved version of this study.”
Evaluation of Text Generation: A Survey (2020, methodology): “Creating high-quality datasets with multiple reference texts is essential for not only improving the reliability of evaluation but also allowing the development of new automatic metrics that correlate well with human judgments (Belz & Reiter, 2006).”
A Snapshot of NLG Evaluation Practices 2005 - 2014 (2015, methodology): “…texts against human written texts for metrics such as quality, correctness, naturalness, understandability, etc. Extrinsic evaluations, on the other hand, have typically consisted of evaluating the impact of a given system such as its effectiveness for a given application (Belz and Reiter, 2006).”
Triple-to-Text: Converting RDF Triples into High-Quality Natural Languages via Optimizing an Inverse KL Divergence (2019, background): “n SemEval dataset. methods MLE CoT PG SeqGAN NW T2T accuracy 0.240 0.258 0.231 0.244 0.155 0.276 5.4 Metrics For natural language generation tasks, the most widely accepted metric is human evaluation [6]. While human evaluation is reliable, it is hardly applied to quality evaluating of large corpus ”
Generating Referring Expressions in Context: The GREC Task Evaluation Challenges (2010, background): “It is not surprising then that stable averages of quality judgements, let alone high levels of agreement, are hard to achieve, as has been observed for mt [36, 22], text summarisation [35], and language generation [2].”
Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task (2020, background): “The recent progress in machine translation models has led researchers to question the use of ngram overlap metrics such as BLEU, which focus solely on surface-level aspects of the generated text, and thus may correlate poorly with human evaluation (Papineni et al., 2002; Lin, 2004; Ma et al., 2019; ”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “The different factors to consider in such an evaluation setup are as follows: Type of evaluators: The evaluators could be experts [10], crowdsourced annotators [15, 61, 130], or even end-users [48, 112] depending on the requirements of the task and the goal of the evaluation.”
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021, methodology): “Each of these metrics is compared to one or more “Gold Standard” text as inspired by the Machine Translation community and adopted for evaluating document summarisation and NLG (Belz and Reiter, 2006).”
Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts (2019, background): “Evaluation has been among the most discussed topics of the natural language generation (NLG) research area (Lapata and Barzilay, 2005; Belz and Reiter, 2006; Reiter and Belz, 2006; Barzi-lay and Lapata, 2008; Reiter and Belz, 2009; Reiter, 2011; Novikova et al., 2017).”
Correlating Human and Automatic Evaluation of a German Surface Realiser (2009, background): “Belz and Reiter (2006) and Reiter and Belz (2009) describe comparison experiments between the automatic evaluation of system output and human (expert and non-expert) evaluation of the same data (English weather forecasts).”