Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation

By Yasuhide Miura, Yuhao Zhang, Curtis P. Langlotz, Dan Jurafsky. 2020

235 citations recorded by the source corpus.

Neural image-to-text radiology report generation systems offer the potential to improve radiology reporting by reducing the repetitive process of report drafting and identifying possible medical errors. However, existing report generation systems, despite achieving high performances on natural language generation metrics such as CIDEr or BLEU, still suffer from incomplete and inconsistent generations. Here we introduce two new simple rewards to encourage the generation of factually complete and consistent radiology reports: one that encourages the system to generate radiology domain entities consistent with the reference, and one that uses natural language inference to encourage these entities to be described in inferentially consistent ways. We combine these with the novel use of an existing semantic equivalence metric (BERTScore). We further propose a report generation system that optimizes these rewards via reinforcement learning. On two open radiology report datasets, our system substantially improved the F1 score of a clinical information extraction performance by +22.1 (Delta +63.9%). We further show via a human evaluation and a qualitative analysis that our system leads to generations that are more factually complete and consistent compared to the baselines.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “Our work is most relevant to work on medical image classification, which we have discussed in Section 1, and textual report generation from medical images (Wang et al., 2018; Jing et al., 2018; Liu et al., 2019; Miura et al., 2021).”
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, methodology): “For report generation, we extend this test set with samples from healthy subjects ( N = 815 ) to match the prevalence of pathological studies used in prior work [14, 25, 45].”
Transformers in Medical Imaging: A Survey (2022, methodology): “[342] propose a transformer-based model that directly optimizes the two newly proposed reward functions using self-critical RL.”
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing (2022, methodology): “[53] Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, and Dan Jurafsky.”