Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View

By Tianyu Liu, Xin Zheng, Baobao Chang, Zhifang Sui. 2021

37 citations recorded by the source corpus.

In open domain table-to-text generation, we notice the unfaithful generation usually contains hallucinated entities which can not be aligned to any input table record. We thus try to evaluate the generation faithfulness with two entity-centric metrics: table record coverage and the ratio of hallucinated entities in text, both of which are shown to have strong agreement with human judgements. Then based on these metrics, we quantitatively analyze the correlation between training data quality and generation fidelity which indicates the potential usage of entity information in faithful generation. Motivated by these findings, we propose two methods for faithful generation: 1) augmented training by incorporating the auxiliary entity information, including both an augmented plan-based model and an unsupervised model and 2) training instance selection based on faithfulness ranking. We show these approaches improve generation fidelity in both full dataset setting and few shot setting by both automatic and human evaluations.

Subfields: Retrieval-Augmented Generation, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Retrieval-Augmented Generation

In Natural Language Processing

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, methodology): “Planning can be a separate step in a two-step generator [22, 107, 163, 182] or be injected into the end-to-end model during generation [200].”
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation (2021, background): “…and template/scaffold guided schema with explicit plans (Ma et al., 2019; Moryossef et al., 2019; Balakrishnan et al., 2019; Du et al., 2020; Liu et al., 2021), e.g. text sequences which specify the narrative ordering, and implicit plans (Wiseman et al., 2018; Ye et al., 2020; Shen et al.,…”
Factuality Enhanced Language Models for Open-Ended Text Generation (2022, background): “…GPT-3 In previous studies, different metrics and methods have been proposed to measure and improve the factual accuracy of language generation within different tasks [11], including text summarization [e.g., 12–15], question answering [e.g., 16–18], and table-to-text generation [e.g., 19, 20].”