Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation

By Clément Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, Patrick Gallinari. 2021

38 citations recorded by the source corpus.

QuestEval is a reference-less metric used in text-to-text tasks, that compares the generated summaries directly to the source text, by automatically asking and answering questions. Its adaptation to Data-to-Text tasks is not straightforward, as it requires multimodal Question Generation and Answering systems on the considered tasks, which are seldom available. To this purpose, we propose a method to build synthetic multimodal corpora enabling to train multimodal components for a data-QuestEval metric. The resulting metric is reference-less and multimodal; it obtains state-of-the-art correlations with human judgment on the WebNLG and WikiBio benchmarks. We make data-QuestEval’s code and models available for reproducibility purpose, as part of the QuestEval project.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…fixed size n-grams, but other basic element metrics like PARENT (Dhingra et al., 2019) only focus on content words. generate questions (Chen et al., 2018; Wang et al., 2020; Durmus et al., 2020; Scialom et al., 2021; Rebuffel et al., 2021; Honovich et al., 2021; Deutsch et al., 2021a, inter alia).”
Survey of Hallucination in Natural Language Generation (2022, methodology): “Data-QuestEval [111] adapt QuestEval [117] from summarization into data-to-text generation.”
QuestEval: Summarization Asks for Fact-based Evaluation (2021, background): “We observe that for all four dimensions and all the baselines, the correlations decrease and the variance increases as the number of references decreases.”
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2021, background): “395 BARTScore (Yuan et al., 2021): Because 396 ROUGE scores only measure token overlap, other 397 automated metrics (Rebuffel et al., 2021; Kryscin- 398 ski et al., 2020; Wang et al., 2020; Scialom et al., 399 2021) have been proposed to evaluate faithfulness400 more precisely.”
A Comprehensive Assessment of Dialog Evaluation Metrics (2021, methodology): “Data-QuestEval(2021) DesignedforgeneralNLU. notspeciﬁcallyfordialog.”