Evaluating Evaluation Methods for Generation in the Presence of Variation

By Amanda Stent, M. Marge, Mohit Singhai. 2005

158 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, neutral): “In contrast, Stent et al. (2005) evaluate metrics on restructured sentences, showing that lexical-overlap based metrics do measure similarity in meaning, but fail at measuring syntactic correctness.”
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation (2016, background): “, 2015) and natural language generation (NLG) (Stent et al., 2005; Cahill, 2009; Reiter and Belz, 2009; Espinosa et al., 2010).”
Why We Need New Evaluation Metrics for NLG (2017, background): “This is rarely the case, as shown by various studies in NLG (Stent et al., 2005; Belz and Reiter, 2006; Reiter and Belz, 2009), as well as in related fields, such as dialogue systems (Liu et al.”
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems (2015, background): “As noted in Stent et al. (2005), a good generator usually depends on several factors: adequacy, ﬂu-ency, readability, and variation.”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, background): “Over the years several studies have evaluated automatic metrics for measuring text generation performance (Callison-Burch et al., 2006; Stent et al., 2005; Belz and Reiter, 2006; Reiter, 2018; Liu et al., 2016; Kilickaya et al., 2017; Gatt and Krahmer, 2018).”
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (2018, background): “…ambiguity when understanding human language (Williams et al., 2013); the need to integrate third-party services and dialogue context in the decision-making (Traum and Lars-son, 2003; Paek and Pieraccini, 2008); and ﬁnally, the ability to generate natural and engaging responses (Stent et al., 2005).”
Summarizing Source Code using a Neural Attention Model (2016, methodology): “Human Evaluation Since automatic metrics do not always agree with the actual quality of the results (Stent et al., 2005), we perform human evaluation studies to measure the output of our system and baselines across two modalities, namely naturalness and informativeness.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Common criteria include: • Fluency or readability, that is, the linguistic quality of the text (e.g., Callaway & Lester, 2002; Mitchell et al., 2012; Stent et al., 2005; Lapata, 2006; Cahill, 2009; Espinosa et al., 2010, inter alia); • Accuracy, adequacy, relevance or correctness relative to the…”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, background): “ and supervised-based methods. 2.1.4 Natural Language Generation The natural language generation component converts an abstract dialogue action into natural language surface utterances. As noticed in [78], a good generator usually relies on several factors: adequacy, ﬂuency, readability, and variati”
Steering Output Style and Topic in Neural Response Generation (2017, neutral): “Owing to the low consistency between automatic metrics and human perception on conversational tasks (Liu et al., 2016; Stent et al., 2005) and the lack of true reference responses from persona models, we evaluated the quality of our generated text with a set of judges recruited from Amazon…”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Coherence is one of the qualities that a good generator seeks (Stent et al., 2005).”
Further Meta-Evaluation of Broad-Coverage Surface Realization (2010, background): “A few other studies have investigated the use of such metrics in evaluating the output of NLG systems, notably (Reiter and Belz, 2009) and (Stent et al., 2005).”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “In particular, despite several studies [3, 17, 150, 177] showing that BLEU and similar metrics do not correlate well with human judgements, there was no decline in their popularity.”
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023, background): “However, these metrics fail to measure content quality (Reiter and Belz, 2009) or capture syntactic errors (Stent et al., 2005), and therefore do not reflect the reliability of NLG systems accurately.”
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020, background): “Despite this, and extensive analytical evidence of the limitations of BLEU in particular and automatic metrics in general (Stent et al., 2005; Callison-Burch and Osborne, 2006; Smith et al., 2016), the metric remains the de facto standard of evaluating research hypotheses.”
Disentangled Representation Learning for Non-Parallel Text Style Transfer (2018, methodology): “Our model is built upon an autoencoder with a sequence-to-sequence neural network (Sutskever, Vinyals, and Le 2014), and we design multi-task and adversarial losses for both style and content spaces.”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, background): “Although more than 60% of recent NLG papers solely use ROUGE or BLEU as the evaluation metric (Kasai et al., 2021), they fail to measure content quality (Reiter and Belz, 2009) and syntactic correctness (Stent et al., 2005), and are thus insufﬁ-cient to portray the reliability of NLG systems.”