A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems

By Craig Thomson, Ehud Reiter. 2020

61 citations recorded by the source corpus.

Most Natural Language Generation systems need to produce accurate texts. We propose a methodology for high-quality human evaluation of the accuracy of generated texts, which is intended to serve as a gold-standard for accuracy evaluations of data-to-text systems. We use our methodology to evaluate the accuracy of computer generated basketball summaries. We then show how our gold standard evaluation can be used to validate automated metrics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Datasets should be cleaned and improved (e.g., Dušek et al., 2019; Thomson and Reiter, 2020) over time and sending pull requests to update data documentation needs to become as commonplace as sending pull requests to or opening issues in open-source libraries.”
Survey of Hallucination in Natural Language Generation (2022, background): “Moreover, the tolerance of hallucination is very low when this task is applied to the real world, such as in the case of patient information table description [181], and analysis of experimental results tables in a scientific report.”
Generation Challenges: Results of the Accuracy Evaluation Shared Task (2021, methodology): “, 2017), which has been widely used by many previous papers to evaluate accuracy, can only detect Name and Number errors and has a recall of less than 40% for these types of errors (Thomson and Reiter, 2020).”
Shared Task on Evaluating Accuracy (2020, methodology): “Our experiences at Aberdeen (Thomson and Reiter, 2020) show that it can take an experienced annotator 30 minutes to factcheck a moderately complex 300-word paragraph produced by a neural data-to-text NLG system.”
Underreporting of errors in NLG output, and what to do about it (2021, background): “The solutions would ideally be created by experts, although in some cases, solutions created by researchers may be sufficient (Thomson and Reiter, 2020).”
Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generation (2021, background): “This is similar to the hallucination problem in summarization and NLG tasks (Thomson and Reiter, 2020).”