An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems

By Ehud Reiter, Anya Belz. 2009

242 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “For instance, Reiter and Belz (2009b) find that these metrics may approximate language quality, although with only weak evidence, and that they do not measure content quality at all.”
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation (2016, background): “, 2015) and natural language generation (NLG) (Stent et al., 2005; Cahill, 2009; Reiter and Belz, 2009; Espinosa et al., 2010).”
Why We Need New Evaluation Metrics for NLG (2017, background): “This is rarely the case, as shown by various studies in NLG (Stent et al., 2005; Belz and Reiter, 2006; Reiter and Belz, 2009), as well as in related fields, such as dialogue systems (Liu et al.”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, background): “Correct semantics is regarded by users as more important than fluency (Reiter and Belz 2009) and should be prioritised when training the models (cf. also Reiter 2019).”
Comparing Automatic Evaluation Measures for Image Description (2014, background): “In contrast to the results presented here, Reiter and Belz (2009) found no signiﬁcant correlations of automatic evaluation measures against human judgements of the accuracy of machine-generated weather forecasts.”
Neural Text Summarization: A Critical Evaluation (2019, methodology): “…the authors put under scrutiny existing methodologies, datasets, or models were conducted by Callison-Burch et al. (2006, 2007); Tan et al. (2015); Post (2018) in machine translation, (Gkatzia and Mahamood, 2015; Reiter and Belz, 2009; Reiter, 2018) in natu-ral language generation, Lee et al. (2016”
A Structured Review of the Validity of BLEU (2018, background): “For example, I excluded Belz and Reiter (2006), since Reiter and Belz (2009) presents an improved version of this study.”
Language Models for Image Captioning: The Quirks and What Works (2015, background): “EOR scores are close for all four models, and that it is well-documented in several NLP communities that BLEU scores do not always correlate with human quality judgments (Callison-Burch et al., 2006; Reiter and Belz, 2009; Hodosh et al., 2013). In future work, we plan to compare human quality judgem”
Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions (2020, background): “…natural language generation (NLG) systems is notoriously complex: the same input can be expressed in a variety of output texts, each valid in its own context, making evaluation with automatic metrics far more challenging than in other NLP contexts (Novikova et al., 2017; Reiter and Belz, 2009).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “…et al., 2007; Cahill & Josef, 2006; Espinosa et al., 2010; Belz et al., 2011) and occasionally also to evaluate short texts characteristic of data-driven systems in domains such as weather reporting (e.g. Reiter & Belz, 2009; Konstas & Lapata, 2013) and image captioning (see Bernardi et al., 2016).”
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (2016, methodology): “Evaluating the output of a natural language generation (NLG) system is a fundamentally difficult task (Dale & White, 2007; Reiter & Belz, 2009).”
Evaluation of Text Generation: A Survey (2020, background): “For example, Young (1999), which Reiter & Belz (2009a) point to as one of the first examples of extrinsic evaluation of generated text, evaluate automatically generated instructions by the number of mistakes subjects made when they followed them.”
A Snapshot of NLG Evaluation Practices 2005 - 2014 (2015, methodology): “For instance, the STOP evaluation cost £ 75,000 over 20 months; the SKILL-SUM and BT45 evaluations cost about £ 20,000 over six months (Reiter and Belz, 2009).”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “Prior work such as Reiter and Belz (2009) high-light the risk of claiming validity without testing generality.”
Further Meta-Evaluation of Broad-Coverage Surface Realization (2010, background): “A few other studies have investigated the use of such metrics in evaluating the output of NLG systems, notably (Reiter and Belz, 2009) and (Stent et al., 2005).”