A Snapshot of NLG Evaluation Practices 2005 - 2014

By Dimitra Gkatzia, Saad Mahamood. 2015

58 citations recorded by the source corpus.

In this paper we present a snapshot of endto-end NLG system evaluations as presented in conference and journal papers1 over the last ten years in order to better understand the nature and type of evaluations that have been undertaken. We find that researchers tend to favour specific evaluation methods, and that their evaluation approaches are also correlated with the publication venue. We further discuss what factors may influence the types of evaluation used for a given NLG system.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “As a result, intrinsic approaches dominate academic publications; Gkatzia and Mahamood (2015) found that about 75% of published NLG systems rely on intrinsic evaluations with the fraction increasing.4 Since we survey widely used approaches, we mostly cover intrinsic evaluations, but stress the…”
Why We Need New Evaluation Metrics for NLG (2017, background): “This result is opposed to the current trend of relying on automatic evaluation identified in (Gkatzia and Mahamood, 2015).”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, background): “Automatic metrics are popular in NLG (Gkatzia and Mahamood 2015) because they are cheaper and faster to run than human evaluation.”
Neural Text Summarization: A Critical Evaluation (2019, methodology): “…the authors put under scrutiny existing methodologies, datasets, or models were conducted by Callison-Burch et al. (2006, 2007); Tan et al. (2015); Post (2018) in machine translation, (Gkatzia and Mahamood, 2015; Reiter and Belz, 2009; Reiter, 2018) in natu-ral language generation, Lee et al. (2016”
RankME: Reliable Human Ratings for Natural Language Generation (2018, background): “Human judgement is the primary evaluation criterion for language generation tasks (Gkatzia and Mahamood, 2015).”
Chain of Explanation: New Prompting Method to Generate Quality Natural Language Explanation for Implicit Hate Speech (2022, background): “Human Evaluation - Informativeness and Clarity Human perceptions have always been considered one of the most important evaluation criteria for Natural Language Generation tasks [9, 12].”
Best practices for the human evaluation of automatically generated text (2019, methodology): “Automatic metrics such as BLEU, METEOR, and ROUGE are increasingly popular; Gkatzia and Mahamood’s (2015) survey of NLG papers from 2005-2014 found that 38.”
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021, background): “Human evaluation is considered a primary evaluation criterion for NLG systems (Gatt and Krahmer, 2018; Mellish and Dale, 1998; Gkatzia and Mahamood, 2015; Hastie and Belz, 2014).”
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, background): “…downstream tasks, from either a user’s perspective (such as whether or not it leads to an improved decision-making or text comprehension accuracy [39], typically through a User Task Success evaluation), or from a system’s perspective [3, 26], such as the performance of a dialogue system through…”
A Human Evaluation of AMR-to-English Generation Systems (2020, background): “Beyond AMR generation, other NLG tasks are also often evaluated only with automatic metrics; for example, Gkatzia and Mahamood (2015) found that 38.2% of NLG papers overall, and 68% of those published in ACL venues, used automatic metrics.”
“This is a Problem, Don’t You Agree?” Framing and Bias in Human Evaluation for Natural Language Generation (2020, background): “the lack of consistency in how human evaluation is carried out (Gkatzia and Mahamood, 2015; van der Lee et al., 2019).”
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications (2022, background): “Despite these and other documented shortcomings (Gkatzia and Mahamood, 2015; Novikova et al., 2017; Kuribayashi et al., 2021; Liang and Li, 2021), practitioners do rely broadly on automatic metrics: 50% of survey participants agree or strongly agree that automatic metrics represent reliable ways to…”
Describing a Knowledge Base (2018, background): “However extrinsic metrics such as system purpose and user task are expensive, while cheaper metrics such as human rating do not correlate with extrinsic metrics (Gkatzia and Mahamood, 2015).”