MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer

By Wei Tang, Fazhi He, Y. Liu, Yansong Duan. IEEE Transactions on Image Processing, 2022

387 citations recorded by the source corpus.

Owing to the limitations of imaging sensors, it is challenging to obtain a medical image that simultaneously contains functional metabolic information and structural tissue details. Multimodal medical image fusion, an effective way to merge the complementary information in different modalities, has become a significant technique to facilitate clinical diagnosis and surgical navigation. With powerful feature representation ability, deep learning (DL)-based methods have improved such fusion results but still have not achieved satisfactory performance. Specifically, existing DL-based methods generally depend on convolutional operations, which can well extract local patterns but have limited capability in preserving global context information. To compensate for this defect and achieve accurate fusion, we propose a novel unsupervised method to fuse multimodal medical images via a multiscale adaptive Transformer termed MATR. In the proposed method, instead of directly employing vanilla convolution, we introduce an adaptive convolution for adaptively modulating the convolutional kernel based on the global complementary context. To further model long-range dependencies, an adaptive Transformer is employed to enhance the global semantic extraction capability. Our network architecture is designed in a multiscale fashion so that useful multimodal information can be adequately acquired from the perspective of different scales. Moreover, an objective function composed of a structural loss and a region mutual information loss is devised to construct constraints for information preservation at both the structural-level and the feature-level. Extensive experiments on a mainstream database demonstrate that the proposed method outperforms other representative and state-of-the-art methods in terms of both visual quality and quantitative evaluation. We also extend the proposed method to address other biomedical image fusion issues, and the pleasing fusion results illustrate that MATR has good generalization capability. The code of the proposed method is available at https://github.com/tthinking/MATR.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/41dd5b992dbbdfcfec809ad7e13da9044d87968b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiao-Jun Wu
2. Yiannis Demiris
3. Xingchen Zhang
4. Hui Li
5. Y. Liu
6. Wei Tang
7. Fazhi He
8. Tongzhen Si

In Media Technology

1. Xiao-Jun Wu
2. Hui Li
3. Yiannis Demiris
4. Xingchen Zhang
5. Wei Tang
6. Y. Liu
7. Fazhi He
8. Tongzhen Si

Citations in the corpus

• CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach (2024, background): “…models [9][10], Bayesian based fusion model [11]. multimodal image fusion also has numerous applications in other fields such as medical diagnosis [12][13], surveillance [14][15], remote sensing [16][17], and robotics [18][19] etc. Deep learning, as a widely popular technology, is no exception in…”
• Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “In 2021, transformers were introduced to the image fusion field, and some transformer-based methods have been proposed to perform VIF [35], [37], [117], [118], [119], [120], [150], [151], general image fusion [34], [36], [121], [152], and other image fusion tasks [153], [154], [155], [156].”
• DATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer (2023, background): “Lately,several Transformer-based fusion models were proposed for other types of image fusion tasks [68], [69], [70].”