MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network

By Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason McGough, Lin F. Yang. 2017

343 citations recorded by the source corpus.

The inability to interpret the model prediction in semantically and visually meaningful ways is a well-known shortcoming of most existing computer-aided diagnosis methods. In this paper, we propose MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process. MDNet includes an image model and a language model. The image model is proposed to enhance multi-scale feature ensembles and utilization efficiency. The language model, integrated with our improved attention mechanism, aims to read and explore discriminative image feature descriptions from reports to learn a direct mapping from sentence words to image pixels. The overall network is trained end-to-end by using our developed optimization strategy. Based on a pathology bladder cancer images and its diagnostic reports (BCIDR) dataset, we conduct sufficient experiments to demonstrate that MDNet outperforms comparative baselines. The proposed image model obtains state-of-the-art performance on two CIFAR datasets as well.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, methodology): “It has been successfully used in modeling multi-level dependencies in image captioning [30, 38], image question answering [31] and machine translation [2].”
Controllable Text-to-Image Generation (2019, background): “The attention mechanism has shown its efficiency in various research fields including image captioning [24, 30], machine translation [1], object detection [14, 29], and visual question answering [26].”
Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network (2018, methodology): “Leveraging hierarchical representations of CNNs is an effective way to enhance implicit multi-scaling and ensembling for tasks such as image recognition [21, 48] and pixel or object classification [41, 25, 22, 37, 44, 50].”
The importance of interpretability and visualization in machine learning for applications in medicine and health care (2019, methodology): “This includes mimicking the performance of DL using simpler more interpretable models such as decision trees [51,52], visual analytics on their own or combined with attention models [53–56], motif analysis [57], or semantic representations [56, 58].”
Thoracic Disease Identification and Localization with Limited Supervision (2017, background): “It is usually formulated as a classification problem where medical imaging abnormalities are identified as different clinical conditions [25, 4, 26, 28, 34].”
Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network (2018, methodology): “Machine learning (ML) based methods have been widely used for medical image analysis [40, 39], including detection, segmentation, and tracking of an anatomical structure.”
DeepMiner: Discovering Interpretable Representations for Mammogram Classification and Explanation (2018, background): “diagnosing bladder pathology images and generating radiological reports if trained on sufficiently large image and diagnostic report datasets [13].”
Density-Aware Single Image De-raining Using a Multi-stream Dense Network (2018, background): “Recently, due to the immense success of deep learning in both high-level and low-level vision tasks [10, 34, 44, 23, 35, 37, 25], several CNN-based methods have also been proposed for image de-raining [4, 6, 36, 7].”
Tell Me Where to Look: Guided Attention Inference Network (2018, unknown): “Since deep neural networks have achieved great success in many areas [7, 34, 35, 37], various methods have been proposed to explain this black box [3, 26, 33, 38].”
Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond (2021, methodology): “As a result, several recent studies [120,123–127] have been devoted to the exploration of explainability in medical AI.”
Deep Learning for Medical Anomaly Detection – A Survey (2020, background): “For instance, in [118] attention is used to map the relationships between medical images and corresponding diagnostic reports.”
Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis (2019, background): “[12, 13] proposed networks generate natural medical report from various Recurrent Neural Networks (RNNs) structure and point informative area of input medical image.”
Explainable Deep Learning Models in Medical Image Analysis (2020, methodology): “ges DeepDreams [46] Custom CNN Segmentation of tumor from liver CT imaging GSInquire, GBP, activation maps [47] COVIDNet CNN COVID-19 detection X-ray images Attention Mapping between image to reports [53] CNN & LSTM Bladder cancer Tissue images U-Net with shape attention stream [54] U-net based ”
Interpretable Sequence Classification Via Prototype Trajectory (2020, background): “Zhang et al. [2017] proposed a language model to read and explore discriminative image feature descriptions from reports to learn a direct mapping between lexical components and image pixels via attention.”