Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports

By Yuhao Zhang, Derek Merck, E. Tsai, Christopher D. Manning, Curtis P. Langlotz. 2019

210 citations recorded by the source corpus.

Neural abstractive summarization models are able to generate summaries which have high overlap with human references. However, existing models are not optimized for factual correctness, a critical metric in real-world applications. In this work, we develop a general framework where we evaluate the factual correctness of a generated summary by fact-checking it automatically against its reference using an information extraction module. We further propose a training strategy which optimizes a neural summarization model with a factual correctness reward via reinforcement learning. We apply the proposed method to the summarization of radiology reports, where factual correctness is a key requirement. On two separate datasets collected from hospitals, we show via both automatic and human evaluation that the proposed approach substantially improves the factual correctness and overall quality of outputs over a competitive neural summarization system, producing radiology summaries that approach the quality of human-authored ones.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “[231] propose to explicitly measure the factual correctness of a generated text against the reference by first extracting facts via an information extraction (IE) module.”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “Other possible ﬁxes include learning task-speciﬁc metrics to measure semantic overlap (Sellam et al., 2020) or metrics to check factual correctness (Zhang et al., 2020) and faithfulness to input (Wang et al., 2020; Durmus et al., 2020; Zhou et al., 2020).”
Evaluation of Text Generation: A Survey (2020, background): “Zhang et al. (2019b) propose a way to tackle the problem of factual correctness in summarization models.”
Factual Error Correction for Abstractive Summarization Models (2020, methodology): “Previous work on factual consistency in abstractive summarization can be divided into two categories: abstractive summarization models tailored towards factual consistency (Cao et al., 2018; Zhang et al., 2019b; Li et al., 2018), and evaluation models for factual consistency in abstractive summariza”
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation (2021, background): “, 2019), information extraction (Zhang et al., 2020; Goodrich et al., 2019) or question answering (Scialom et al.”
Enhancing Factual Consistency of Abstractive Summarization (2021, methodology): “Zhang et al. (2019a) employs BERT to compute similarity between pairs of words in the summary and article.”
Reducing the Frequency of Hallucinated Quantities in Abstractive Summaries (2020, methodology): “Zhang et al. (2019) developed a framework to evaluate the factual correctness of generated summaries by employing an information extraction module to check facts against the source document, and proposed a training strategy that optimizes the model using reinforcement learning with factual…”
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation (2022, unknown): “, 2016b; Alsentzer and Kim, 2018), and with abstractive neural methods (Zhang et al., 2018; Liu et al., 2019; MacAvaney et al., 2019; Zhang et al., 2020; Enarvi et al., 2020; Joshi et al., 2020; Krishna et al., 2021; Chintagunta et al., 2021; Yim and Yetisgen-Yildiz, 2021; Moramarco et al., 2021; Zh”
The Summary Loop: Learning to Write Abstractive Summaries Without Examples (2020, methodology): “Recent work around measuring factuality in generated text, using Natural Language Inference (Guo et al., 2018) or rule-based fact extraction (Zhang et al., 2019b) becomes increasingly important with summaries that are more abstractive.”
Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization (2021, methodology): “(1) Employ information extraction tools to extract facts and leveraging it by building additional ob-jective (Cao et al., 2018; Goodrich et al., 2019; Zhang et al., 2020a; Zhu et al., 2021).”
Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques (2020, background): “The detection and correction of such factual errors in automatically generated summaries is an active area of research (Cao et al., 2018; Zhang et al., 2020; Dong et al., 2020).”
Don’t Say What You Don’t Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search (2022, methodology): “This includes models like BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020a), and UniLM (Dong et al., 2019).”
Annotating and Modeling Fine-grained Factuality in Summarization (2021, background): “The generation models rarely makes extrinsic hallucinations, and we observed that these are even less common in recent systems like PEGASUS (Zhang et al., 2020a).”
Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View (2021, background): “…human (Chen and Bansal 2018) or automatic metrics using natural language inference (Kry´sci´nski et al. 2019; Falke et al. 2019), information extraction (Zhang et al. 2019; Goodrich et al. 2019) or question answering (Scialom et al. 2019; Eyal, Baumel, and Elhadad 2019; Wang, Cho, and Lewis 2020).”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, background): “…performance on abstractive summarization (Liu and Lapata, 2019; Lewis et al., 2020; Zhang et al., 2020a) with impeccable ﬂuency, yet their summaries often contain factually inconsistent content (Maynez et al., 2020; Zhang et al., 2020b; Goyal and Durrett, 2020), even for state-of-the-art models.”