Should Corpora Texts Be Gold Standards for NLG?

By Ehud Reiter, Somayajulu Gowri Sripada. 2002

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “To the extent possible, evaluation reports should be framed in causal terms by measuring the response of multiple metrics (or human evaluation) to stimuli to avoid issues with metrics, similar to the CheckList framework by Ribeiro et al. (2020). This has the further advantage of setting more realist”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Corpora often display a substantial range of variation and it is often unclear, without an independent assessment, which outputs are to be preferred (Reiter & Sripada, 2002).”
A Snapshot of NLG Evaluation Practices 2005 - 2014 (2015, background): “In addition, Reiter and Sripada (2002) argue that the information presented in aligned corpora might not be always true, due to the fact that text from experts can be erroneous.”
Individual and Domain Adaptation in Sentence Planning for Dialogue (2007, background): “• An examination of a weather report corpus for five weather forecasters showed individual differences in lexical choice for expressing specific weather-related concepts (Reiter & Sripada, 2002).”
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (2019, unknown): “First, these automatic evaluations rely on a high-quality reference corpus as references, which is not often available; variations in the writing style of authors and the presence of (grammatical) errors often result in inconsistent and error prone judgments (Reiter and Sripada, 2002).”
An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems (2009, methodology): “this is an intrinsic form of evaluation (Spärck Jones and Galliers 1995). This methodology was first used in NLG by Lester and Porter (1997), who asked eight domain experts to each rate 15 texts on a number of different dimensions: overall quality and coherence, content, organization, writing style,”