Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation

By Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner-Fushman, Jianhua Yao, Ronald M. Summers. 2016

394 citations recorded by the source corpus.

Despite the recent advances in automatically describing image contents, their applications have been mostly limited to image caption datasets containing natural images (e.g., Flickr 30k, MSCOCO). In this paper, we present a deep learning model to efficiently detect a disease from an image and annotate its contexts (e.g., location, severity and the affected organs). We employ a publicly available radiology dataset of chest x-rays and their reports, and use its image annotations to mine disease names to train convolutional neural networks (CNNs). In doing so, we adopt various regularization techniques to circumvent the large normalvs-diseased cases bias. Recurrent neural networks (RNNs) are then trained to describe the contexts of a detected disease, based on the deep CNN features. Moreover, we introduce a novel approach to use the weights of the already trained pair of CNN/RNN on the domain-specific image/text dataset, to infer the joint image/text contexts for composite image labeling. Significantly improved image annotation results are demonstrated using the recurrent neural cascade model by taking the joint image/text contexts into account.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

A survey on deep learning in medical image analysis (2017, background): “…leveraging reports to improve image classiﬁcation accuracy (Schlegl et al., 2015), and (2) generating text reports from images (Kisilev et al., 2016; Shin et al., 2015, 2016a; Wang et al., 2016e); the latter inspired by recent caption generation papers from natural images (Karpa-thy and Fei-Fei,…”
ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases (2017, methodology): “This database is utilized in [41] as a problem of caption generation but no quantitative disease detection results are reported.”
Deep Collaborative Embedding for Social Image Understanding (2019, background): “In contrast to traditional image annotation methods [21], [22], [23], [24], [25], [26] and image captioning [20], [27], social image tagging is to remove the noisy or irrelevant”
Abnormality Detection and Localization in Chest X-Rays using Deep Convolutional Neural Networks (2017, methodology): “We have found the existing literature to be insufficient for making comparison of various detection techniques either due to studies reported on private datasets or not reporting the test scores in proper detail [13].”
Deep Learning Applications in Medical Image Analysis (2018, background): “[31] describe annotating X-ray images with captions trained on radiology reports.”
Deep Learning in Medical Imaging: General Overview (2017, background): “…have recently introduced the use of CNNs and RNNs (60-68) to combine recent advances in computer vision and machine translation, and thus automatically annotate chest radiographs with diseases and descriptions of the context of a disease (e.g., the location, severity, and the affected organs) (81).”
Efficient Deep Network Architectures for Fast Chest X-Ray Tuberculosis Screening and Visualization (2019, methodology): “Since the network is able to localize the symptoms of tuberculosis, it could even be possible to generate a textual annotation for each case, as was successfully done in similar publications 18 .”
Shifting machine learning for healthcare from development to deployment and from models to data (2022, methodology): “NLP has enabled the automated translation of doctor–patient interactions to notes 5,104,105 , the summarization of clinical notes 106 , the captioning of medical images 107,108 and the prediction of disease progression 6,7 .”
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network (2017, background): “To date, [25] and [17] deal with the problem of generating disease keywords for radiology images.”
Clinically Accurate Chest X-Ray Report Generation (2019, background): “In the ﬁeld of radiology, Shin et al. (2016) proposed a convolutional and recurrent network framework that jointly trained from image and text to annotate disease, anatomy, and severity in the chest X-ray images.”
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (2019, background): “Chest X-ray Image Report Generation Shin et al. (2016) ﬁrst proposed a variant of CNN-RNN framework to predict tags (location and severity) of chest X-ray images.”
Learning to diagnose from scratch by exploiting dependencies among labels (2017, background): “However, they perform training and inference in the manner of Shin et al. (2016).”
Deep Learning in Medical Image Analysis (2017, background): “…vision inspired its use to medical image analysis such as image segmentation (22, 23), image registration (24), image fusion (25), image annotation (26), computer-aided diagnosis and prognosis (27, 28, 29), lesion/landmark detection (30, 31, 32), and microscopic imaging analysis (33, 34), to name…”
TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays (2018, background): “In medical imaging domain, Shin et al.[32] proposed to correlate the entire image or saliency regions with MeSH terms.”
Thoracic Disease Identification and Localization with Limited Supervision (2017, background): “It is usually formulated as a classification problem where medical imaging abnormalities are identified as different clinical conditions [25, 4, 26, 28, 34].”