Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference
By Ondrej Dusek, Zdeněk Kasner. 2020
80 citations recorded by the source corpus.
A major challenge in evaluating data-to-text (D2T) generation is measuring the semantic accuracy of the generated text, i.e. checking if the output text contains all and only facts supported by the input data. We propose a new metric for evaluating the semantic accuracy of D2T generation based on a neural model pretrained for natural language inference (NLI). We use the NLI model to check textual entailment between the input data and the output text in both directions, allowing us to reveal omissions or hallucinations. Input data are converted to text for NLI using trivial templates. Our experiments on two recent D2T datasets show that our metric can achieve high accuracy in identifying erroneous system outputs.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Information Systems
- Dilek Z. Hakkani-Tür
- Tiezheng Yu
- Nayeon Lee
- Yang Liu
- Pascale Fung
- Wenliang Dai
- Sashank Santhanam
- Michael Castelle
In Natural Language Processing
- Pascale Fung
- Andrea Madotto
- Nayeon Lee
- Etsuko Ishii
- Yejin Bang
- Yang Liu
- Dilek Z. Hakkani-Tür
- Tiezheng Yu
Citations in the corpus
- Survey of Hallucination in Natural Language Generation (2022, background): “[172] Data2Text Dušek and Kasner [39], Liu et al.”
- Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generation (2021, background): “This is similar to the hallucination problem in summarization and NLG tasks (Thomson and Reiter, 2020).”