On the Automatic Generation of Medical Imaging Reports

By Baoyu Jing, Pengtao Xie, Eric P. Xing. 2017

673 citations recorded by the source corpus.

Medical imaging is widely used in clinical practice for diagnosis and treatment. Report-writing can be error-prone for unexperienced physicians, and time-consuming and tedious for experienced physicians. To address these issues, we study the automatic generation of medical imaging reports. This task presents several challenges. First, a complete report contains multiple heterogeneous forms of information, including findings and tags. Second, abnormal regions in medical images are difficult to identify. Third, the reports are typically long, containing multiple sentences. To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the prediction of tags and the generation of paragraphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs. We demonstrate the effectiveness of the proposed methods on two publicly available dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “Our work is most relevant to work on medical image classification, which we have discussed in Section 1, and textual report generation from medical images (Wang et al., 2018; Jing et al., 2018; Liu et al., 2019; Miura et al., 2021).”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “We compared VisualGPT against state-of-the-art medical report generation models including Att2in [61], CoAtt [26], HRGR [35], CMAS-RL [25] and the model from Chen et al. [10].”
Generative Adversarial Network in Medical Imaging: A Review (2018, neutral): “Tang et al. (2018) SGAN L1, 2, 8 – M32 [ √ ] Denoising, contrast enhance Shan et al.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, background): “In an early stage, the study [117] proposed a data-driven neural network that combines a convolutional neural network with an LSTM to predict medical tags and generate a single sentence report, by employing a co-attention mechanism over visual and textual features.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “imitate the style of real medical reports [227], [228], [229].”
Secure and Robust Machine Learning for Healthcare: A Survey (2020, background): “A multi-task ML based framework is proposed for automatic tagging and description of medical images [63].”
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation (2018, methodology): “On IU X-Ray dataset, we also compare with CoAtt [16] which uses different visual features extracted from a pretrained ResNet [11].”
Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (2019, methodology): “Automatic generation of medical image reports, as one of the key applications in the ﬁeld, is gaining increasing research interest (Li et al. 2018; Wang et al. 2018b; Jing, Xie, and Xing 2018).”
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (2019, methodology): “Comparison Methods For Findings section, we compare our proposed method with state-of-the-art methods for CXR imaging report generation: CoAtt (Jing et al., 2018) and HGRG-Agent (Li et al., 2018), as well as several state-of-the-art image captioning models: CNN-RNN (Vinyals et al., 2015), LRCN (Dona”
PadChest: A large chest x-ray image dataset with multi-label annotated reports (2019, methodology): “Chest-XRay14 was also employed by [7] to design a network architecture that combines text and image with attention mechanisms capable of generating text that describes the image, while [8] introduced a hierarchical model of Recurrent Neural Networks (RNN) with which to generate long paragraphs from ”
EEGtoText: Learning to Write Medical Reports from EEG Recordings (2019, background): “Although there are a number of image/video captioning (Vincent et al., 2008; Xu et al., 2015b) or medical image reporting generation methods (Jing et al., 2017b; Li et al., 2018) being proposed, they tend to perform poorly on clinical time series report generation due to two primary reasons.”
DeepMiner: Discovering Interpretable Representations for Mammogram Classification and Explanation (2018, background): “Jing et al. (2017) successful created a visual and semantic network that directly generates long-form radiological reports for chest X-rays after training on a dataset of X-ray images and associated ground truth reports.”
A generalist vision–language foundation model for diverse biomedical tasks (2023, positive): “On the Peir Gross dataset, our BiomedGPT model surpassed the existing SOTA benchmark 36 , demonstrating improvements of 8.1 percentage points in ROUGE-L and 0.5 points in METEOR, and a substantial gain of 89.8 points in the CIDEr metric.”