Handling Divergent Reference Texts when Evaluating Table-to-Text Generation

By Bhuwan Dhingra, Manaal Faruqui, Ankur P. Parikh, Ming-Wei Chang, Dipanjan Das, William W. Cohen. 2019

225 citations recorded by the source corpus.

Automatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge. We propose a new metric, PARENT, which aligns n-grams from the reference and generated texts to the semi-structured data before computing their precision and recall. Through a large scale human evaluation study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics. We also adapt and evaluate the information extraction based evaluation proposed by Wiseman et al (2017), and show that PARENT has comparable correlation to it, while being easier to use. We show that PARENT is also applicable when the reference texts are elicited from humans using the data from the WebNLG challenge.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Instantiations of these metrics may blank out entities (Eyal et al., 2019; Xie et al., 2021; Scialom et al., 2019), or fully 5ROUGE is a special case of this where basic elements are fixed size n-grams, but other basic element metrics like PARENT (Dhingra et al., 2019) only focus on content words. g”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “grained and interpretable evaluation metrics, for example to measure consistency in data-to-text problems (Opitz and Frank, 2020; Dhingra et al., 2019).”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “A model with a poorly-informed decoder and that is agnostic to the divergence issue between the source and target texts (Wiseman et al., 2017; Dhingra et al., 2019), will function more as an open-ended language model and will be prone to extrinsic hallucinations.”
Survey of Hallucination in Natural Language Generation (2022, methodology): “PARENT [22] measures the accuracy of table-to-text generation by aligning n-grams from the reference description R and generated texts G to the table T .”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “Furthermore, we include PARENT (Dhingra et al., 2019) which explicitly aligns n-grams from the reference and generated text to the data contents.”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, background): “…BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), TER (Snover et al., 2005), MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020) and PARENT (Dhingra et al., 2019”
What Makes Good In-Context Examples for GPT-3? (2021, positive): “The KATE method gives rise to considerable gains over the random base-line, according to both the BLEU and PARENT scores.”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, methodology): “PARENT (Dhingra et al., 2019): A metric recently proposed speciﬁcally for data-to-text evaluation that takes the table into account.”
Text-to-Text Pre-Training for Data-to-Text Tasks (2020, neutral): “Dhingra et al. (2019) find that PARENT correlates better with human factual accuracy judgements in comparison to other generation metrics like ROGUE (Lin, 2004) and METEOR.”
Planning with Learned Entity Prompts for Abstractive Summarization (2021, background): “Basically, our results demonstrate that when models are trained on the noisy data (with data divergence issues, common in summarization datasets (Dhingra et al., 2019; Maynez et al., 2020)), the drop-prompt mechanism is very effective in generating high quality summaries when entities need to be sim”
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets (2021, background): “Contrary to this expectation, models obtain lower precision scores on SynthBio than WikiBio as measured by PARENT-P and ROUGE, shown in Table 6.”
Plan-then-Generate: Controlled Data-to-Text Generation via Planning (2021, methodology): “We report the automatic result of BLEU-4, PARENT3 (Dhingra et al., 2019), and a learnt metric BLEURT (Sellam et al.”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, background): “The invariant and ﬂuency-based (Dhingraetal.,2019) Figure 2: Metrics analysed in this study templates are common for all the tasks considered in this work.”
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports (2019, background): “While our work is focused on improving neural summarization models, we note that the idea of using information extraction systems to evaluate the ﬁdelity of generated text has also been explored for data-to-text generation (Wiseman et al., 2017; Dhingra et al., 2019).”
Is ChatGPT a Good NLG Evaluator? A Preliminary Study (2023, background): “Besides, a specifically developed metric named PARENT (Dhingra et al., 2019) is designed for the table2text generation.”
Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation (2019, methodology): “Furthermore, many previous works rely on automatic metrics such as BLEU, which can be poorly correlated with human judgment of faithfulness (Wiseman et al., 2017; Dhingra et al., 2019).”
Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation (2021, methodology): “Based on n-grams similarity between an evaluated text and its gold references, BLEU (Papineni et al., 2002) and PARENT (Dhingra et al., 2019) are the two standard metrics reported in DTG evaluations.”
Logic2Text: High-Fidelity Natural Language Generation from Logical Forms (2020, background): “More recent works begin to focus on fidelity preserving of the generation, such as (Dhingra et al., 2019; Tian et al., 2019).”