TransMed: Transformers Advance Multi-Modal Medical Image Classification

By Yin Dai, Yifan Gao. Diagnostics, 2021

460 citations recorded by the source corpus.

Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it cannot deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependencies, and effective multi-modal fusion strategies can greatly improve the performance of deep models. This prompts us to study transformer-based structures and apply them to multi-modal medical images. Existing transformer-based network architectures require large-scale datasets to achieve better performance. However, medical imaging datasets are relatively small, which makes it difficult to apply pure transformers to medical image analysis. Therefore, we propose TransMed for multi-modal medical image classification. TransMed combines the advantages of CNN and transformer to efficiently extract low-level features of images and establish long-range dependencies between modalities. We evaluated our model on two datasets, parotid gland tumors classification and knee injury classification. Combining our contributions, we achieve an improvement of 10.1% and 1.9% in average accuracy, respectively, outperforming other state-of-the-art CNN-based models. The results of the proposed method are promising and have tremendous potential to be applied to a large number of medical image analysis tasks. To our best knowledge, this is the first work to apply transformers to multi-modal medical image classification.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/03e13ef9192206fecdb227366b298c992dbf7061

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Mahmut Yurt
2. Tolga Cukur
3. Qian Wang
4. Dinggang Shen
5. Zihao Yin
6. Wen Ji
7. Junfeng Zhang
8. Onat Dalmaz

In Radiology, Nuclear Medicine and Imaging

1. Dinggang Shen
2. Qian Wang
3. Islem Rekik

Citations in the corpus

• ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis (2021, background): “While attention maps might be distributed across image regions, multiplicative gating of local CNN features offers limited expressiveness in modeling of global context [51], [89], [90].”
• Transformers in Medical Image Analysis: A Review (2022, methodology): “DRT [58] Glaucoma Eye ZOC-OCT&VF* use two relation module to extract intermodal relation; use Trans to fuse features Diagnosis TransMed [142] Parotid Gland Tumor Parotid Gland * use Trans to capture cross-modality mutual information and fuse features Prediction”
• A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications (2023, methodology): “A model is applied for learning representation between target variable and input instances [19, 20].”