Evaluation Metrics for Generation

By Srinivas Bangalore, Owen Rambow, S. Whittaker. 2000

139 citations recorded by the source corpus.

Certain generation applications may profit from the use of stochastic methods. In developing stochastic methods, it is crucial to be able to quickly assess the relative merits of different approaches or models. In this paper, we present several types of intrinsic (system internal) metrics which we have used for baseline quantitative assessment. This quantitative assessment should then be augmented to a fuller evaluation that examines qualitative aspects. To this end, we describe an experiment that tests correlation between the quantitative metrics and human qualitative judgment. The experiment confirms that intrinsic metrics cannot replace human evaluation, but some correlate significantly with human judgments of quality and understandability and can be used for evaluation during development.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “These include accuracy of parse trees (Bangalore et al., 2000), overlap between “basic elements” (Hovy et al.”
Minimum Error Rate Training in Statistical Machine Translation (2003, methodology): “Examples of such methods are word error rate, position-independent word error rate (Tillmann et al., 1997), generation string accuracy (Bangalore et al., 2000), multi-reference word error rate (Nießen et al., 2000), BLEU score (Pap-ineni et al., 2001), NIST score (Doddington, 2002).”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, background): “Hence, automatic metrics that compare the generated text to one or more reference texts are routinely used to compare models (Bangalore et al., 2000).”
UvA-DARE (Digital Academic Repository) A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs (unknown): “…accuracy of individual word or constituent deletion decisions; Simple String Accuracy (string edit distance), introduced by Banga-lore et al. (2000) for natural language generation tasks; and Word Accuracy (Chiori and Furui, 2004), which generalizes Bangalore et al. (2000) to multiple references.”
Controlling Personality-Based Stylistic Variation with Neural Natural Language Generators (2018, background): “It is widely agreed that new evaluation metrics are needed for NLG (Langkilde-Geary, 2002; Belz and Reiter, 2006; Bangalore et al., 2000; Novikova et al., 2017a).”
The TUNA-REG Challenge 2009: Overview and Evaluation Results (2009, methodology): “We also used the version of string-edit distance described by Bangalore et al. (2000) which normalises for length.”
Comparing Automatic and Human Evaluation of NLG Systems (2006, background): “…judgments (Papineni et al., 2002; Doddington, 2002), we are not aware of any studies that have shown that corpus-based evaluation metrics of NLG systems are correlated with human judgments; correlation studies have been made of individual components (Bangalore et al., 2000), but not of systems.”
Natural language processing: state of the art, current trends and challenges (2017, methodology): “, 2000) [8], multi-reference word error rate (Nießen et al.”
Intrinsic vs. Extrinsic Evaluation Measures for Referring Expression Generation (2008, methodology): “We also used the version of string-edit distance (‘ SEB ’) of Bangalore et al. (2000) which normalises for length.”
Evaluating Evaluation Methods for Generation in the Presence of Variation (2005, methodology): “Only SSA and BLEU have previously been used to evaluate the output of generation systems; SSA, BLEU, NIST and the F measure are designed for the evaluation of machine translation output.”
Global inference for sentence compression : an integer linear programming approach (2008, methodology): “Although automatic evaluation measures have been proposed (Riezler et al., 2003; Bangalore, Rambow, & Whittaker, 2000 ) their use is less widespread, we suspect due to the small size of the test portion of the Ziff-Davis corpus which is commonly used in compression work.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “They found that BLEU, NIST, Simple String Accuracy (SSA) [8], Melamed’s F-measure (i.”
Computational Generation of Referring Expressions: A Survey (2012, background): “…string distance metric, which has also been proposed for REG evaluation, is the Levenshtein (1966) distance: The minimal number of insertions, deletions, and substitutions needed to convert one string into another, possibly normalized with respect to length (Bangalore, Rambow, and Whittaker 2000).”
An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems (2009, methodology): “The ﬁrst such study we are aware of is Bangalore, Rambow, and Whittaker (2000), who looked at string-edit and tree-edit metrics (this work predates BLEU and ROUGE ) using a small number of manually simulated system “outputs.””