A Semantic QA-Based Approach for Text Summarization Evaluation

By Ping Chen, Fei Wu, Tong Wang. 2017

43 citations recorded by the source corpus.

Many Natural Language Processing and Computational Linguistics applications involve the generation of new texts based on some existing texts, such as summarization, text simplification and machine translation. However, there has been a serious problem haunting these applications for decades, that is, how to automatically and accurately assess quality of these applications. In this paper, we will present some preliminary results on one especially useful and challenging problem in NLP system evaluation---how to pinpoint content differences of two text passages (especially for large passages such as articles and books). Our idea is intuitive and very different from existing approaches. We treat one text passage as a small knowledge base, and ask it a large number of questions to exhaustively identify all content points in it. By comparing the correctly answered questions from two text passages, we will be able to compare their content precisely. The experiment using 2007 DUC summarization corpus clearly shows promising results.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Choi and Pak (2005) provide a longer catalogue covering 48 of these biases. However, if researchers do not report the details of their studies, no one can judge whether these biases apply; surveys of NLG papers find as few as 35% (Howcroft et al., 2020) and 16% (Schoch et al., 2020) of papers share ”
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models (2019, methodology): “• Extending recent works (Eyal et al., 2019; Chen et al., 2018), we introduce new metrics, based on Question Answering, that do not require human annotations.”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, methodology): “Recently, question answering (QA) based automatic metrics have been proposed for evaluating content selection in summarization (Eyal et al., 2019; Scialom et al., 2019; Chen et al., 2018).”
QuestEval: Summarization Asks for Fact-based Evaluation (2021, methodology): “Chen et al. (2017) and Eyal et al. (2019) proposed to automate this approach by automatically generating questions from the reference summary.”
Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation (2021, background): “A recent approach, QuestEval (Scialom et al., 2021) has shown signiﬁcant improvement over standard metrics on Summarization tasks.”
SAFEval: Summarization Asks for Fact-based Evaluation (2021, methodology): “Chen et al. (2017) and Eyal et al. (2019) proposed to automate this approach by automatically generating questions from the reference summary.”
TuringAdvice: A Generative and Dynamic Evaluation of Language Use (2021, methodology): “Recent work uses pretrained question-answering models to evaluate summarization models (Chen et al., 2018; Scialom et al., 2019; Eyal et al., 2019; Vasilyev et al., 2020).”
Evaluating Machines by their Real-World Language Use (2020, methodology): “Recent work uses pretrained question-answering models to evaluate summarization models (Chen et al., 2018; Scialom et al., 2019; Eyal et al., 2019; Vasilyev et al., 2020).”
Fill in the BLANC: Human-free quality estimation of document summaries (2020, background): “For example this might be achieved through a series of question-answers [16, 17, 18].”