TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays

By Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Ronald M. Summers. 2018

560 citations recorded by the source corpus.

Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Multimodal Learning

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “Our work is most relevant to work on medical image classification, which we have discussed in Section 1, and textual report generation from medical images (Wang et al., 2018; Jing et al., 2018; Liu et al., 2019; Miura et al., 2021).”
Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (2019, methodology): “Automatic generation of medical image reports, as one of the key applications in the ﬁeld, is gaining increasing research interest (Li et al. 2018; Wang et al. 2018b; Jing, Xie, and Xing 2018).”
Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays (2021, methodology): “TieNet [39] represents the state-of-the-art method in the ﬁeld, which is also based on ResNet-50 and a LSTM based text encoder. each disease class and Average (AVG) AUCs for all models.”
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (2021, methodology): “I: Following [42, 44, 12, 14, 40, 29], we adopt the ResNet-152 [11] to extract 2,048 7× 7 image feature maps which are further projected into 512 7× 7 feature maps, resulting I = {i1, i2, . . . , iNI} ∈ RNI×d (NI = 49, d = 512).”
Clinically Accurate Chest X-Ray Report Generation (2019, methodology): “Later on, Wang et al. (2018) used the same Chest X-ray dataset to further improve the performance of disease classiﬁcation and report generation from an image.”
Deep convolutional neural network based medical image classification for disease diagnosis (2019, background): “In the two general resolving directions, one is to collect more data, such as crowdsourcing [5] or digging into the existing clinical reports [6].”
Shifting machine learning for healthcare from development to deployment and from models to data (2022, methodology): “NLP has enabled the automated translation of doctor–patient interactions to notes 5,104,105 , the summarization of clinical notes 106 , the captioning of medical images 107,108 and the prediction of disease progression 6,7 .”
Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images (2018, unknown): “In the context of medical image analysis, attention models have been exploited for medical report generation (Zhang et al., 2017a,b) as well as joint image and text classification (Wang et al., 2018).”
Deep learning in generating radiology reports: A survey (2020, unknown): “Often RNNs are accompanied with CNNs to generate medical image descriptions [24,27,51,53] (Fig.”
Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis (2019, background): “[22, 24] proposed the networks generating natural medical reports from various Recurrent Neural Networks (RNNs) structure and pointing an informative area of input medical image.”
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (2019, methodology): “Existing methods (Wang et al., 2018a; Jing et al., 2018; Li et al., 2018) ignored the relationship between Findings and Impression , as well as the different distributions between normal and abnormal ﬁndings/conclusions.”
When Radiology Report Generation Meets Knowledge Graph (2020, methodology): “We also included Several previous works for comparison, i.e. ChestXray8 (Wang et al. 2017) and TieNet (Wang et al. 2018).”
PadChest: A large chest x-ray image dataset with multi-label annotated reports (2019, methodology): “Chest-XRay14 was also employed by [7] to design a network architecture that combines text and image with attention mechanisms capable of generating text that describes the image, while [8] introduced a hierarchical model of Recurrent Neural Networks (RNN) with which to generate long paragraphs from ”
Review of Artificial Intelligence Techniques in Imaging Data Acquisition, Segmentation, and Diagnosis for COVID-19 (2020, background): “3) Multidisciplinary integration, i.e., medical imaging [85] , natural language processing [86] , and oncology and fusion [86] , could benefit the overall follow-up procedure of measurement for COVID-19.”