Clinically Accurate Chest X-Ray Report Generation

By Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew B. A. McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, Marzyeh Ghassemi. 2019

353 citations recorded by the source corpus.

The automatic generation of radiology reports given medical radiographs has significant potential to operationally and improve clinical patient care. A number of prior works have focused on this problem, employing advanced methods from computer vision and natural language generation to produce readable reports. However, these works often fail to account for the particular nuances of the radiology domain, and, in particular, the critical importance of clinical accuracy in the resulting generated reports. In this work, we present a domain-aware automatic chest X-ray radiology report generation system which first predicts what topics will be discussed in the report, then conditionally generates sentences corresponding to these topics. The resulting system is fine-tuned using reinforcement learning, considering both readability and clinical accuracy, as assessed by the proposed Clinically Coherent Reward. We verify this system on two datasets, Open-I and MIMIC-CXR, and demonstrate that our model offers marked improvements on both language generation metrics and CheXpert assessed accuracy over a variety of competitive baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Pulmonary and Respiratory Medicine, Radiology, Nuclear Medicine and Imaging, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Pulmonary and Respiratory Medicine

Xiaosong Wang

In Radiology, Nuclear Medicine and Imaging

In Natural Language Processing

Citations in the corpus

Release Strategies and the Social Impacts of Language Models (2019, background): “These include biomedical literature analysis [4], generating synthetic test data [27], and generating radiology reports [37] and EEG reports [ 7].”
Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “Our work is most relevant to work on medical image classification, which we have discussed in Section 1, and textual report generation from medical images (Wang et al., 2018; Jing et al., 2018; Liu et al., 2019; Miura et al., 2021).”
When Radiology Report Generation Meets Knowledge Graph (2020, methodology): “Liu et al. (Liu et al. 2019) applied self-critical sequence training (Rennie et al. 2017) based on reinforcement learning to optimize a clinically coherent reward, which focuses on the correct mention of disease keywords.”
Generating Radiology Reports via Memory-driven Transformer (2020, methodology): “Liu et al. (2019) proposed an approach with reinforcement learning to maintain the clinical accuracy of generated reports.”
Cross-modal Memory Networks for Radiology Report Generation (2022, background): “Therefore, it is expected to have a better solution to model the alignments across modalities and further improve the generation ability, although promising results are continuously acquired by other approaches (Li et al., 2018; Liu et al., 2019; Jing et al., 2019; Chen et al., 2020).”
Automatic Detection of Machine Generated Text: A Critical Survey (2020, methodology): “…used to generate text that approximately matches the style of human language, which beneﬁts applications such as story generation (Fan et al., 2018), conversational response generation (Zhang et al., 2020), code auto-completion (TabNine, 2020), and radiology report generation (Liu et al., 2019a).”
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports (2019, background): “Li et al. (2019) and Liu et al. (2019) studied generating textual descriptions of radiology ﬁndings from medical images, and proposed RL-based approaches to tackle this problem.”
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing (2022, background): “[45] Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi.”