Intrinsic vs. Extrinsic Evaluation Measures for Referring Expression Generation

By Anya Belz, Albert Gatt. 2008

54 citations recorded by the source corpus.

In this paper we present research in which we apply (i) the kind of intrinsic evaluation metrics that are characteristic of current comparative HLT evaluation, and (ii) extrinsic, human task-performance evaluations more in keeping with NLG traditions, to 15 systems implementing a language generation task. We analyse the evaluation results and find that there are no significant correlations between intrinsic and extrinsic evaluation measures for this task.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “For instance, Belz and Gatt (2008) investigate the correlation between lexical overlap metrics (such as BLEU and ROUGE) and various measures of success in a Referring Expression Generation context.”
A Structured Review of the Validity of BLEU (2018, background): “Indeed only one paper correlated system-level BLEU scores with task outcomes (Belz and Gatt 2008), and all such correlations in this paper were Low or Negative.”
An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems (2009, background): “For example Belz and Gatt (2008) analyzed correlations between several automatic evaluation metrics and task performance in a referring-expression generation task; they found that there was no signiﬁcant correlation between any of the automatic metrics they looked at (which included specialized…”
The aNALoGuE Challenge: Non Aligned Language GEneration (2016, background): “” Previous research has shown that automatic metrics like BLEU do not consistently agree with human perception (Stent et al., 2004; Belz and Gatt, 2008).”
The TUNA-REG Challenge 2009: Overview and Evaluation Results (2009, result): “While these results differ from previous ﬁndings (Belz and Gatt, 2008), in which no signiﬁcant correlations were found between extrinsic measures and automatic intrinsic metrics, it is worth noting that signiﬁcance in the results reported here was only observed between human-assessed intrinsic…”
Computational Generation of Referring Expressions: A Survey (2012, background): “Generation of referring expressions in a text is studied in the GREC (Generating Referring Expressions in Context) challenges (Belz et al. 2008).”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, background): “Belz and Gatt (2008) referred to this similarity as “human-likeness” and pointed out that the ability to output human-like text may be completely unrelated to the ﬁnal performance on generation tasks.”