SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer

By Jiayi Ma, Linfeng Tang, Fan Fan, Jun Huang, Xiaoguang Mei, Yong Ma. IEEE/CAA Journal of Automatica Sinica, 2022

1,252 citations recorded by the source corpus.

This study proposes a novel general image fusion framework based on cross-domain long-range learning and Swin Transformer, termed as SwinFusion. On the one hand, an attention-guided cross-domain module is devised to achieve sufficient integration of complementary information and global interaction. More specifically, the proposed method involves an intra-domain fusion unit based on self-attention and an inter-domain fusion unit based on cross-attention, which mine and integrate long dependencies within the same domain and across domains. Through long-range dependency modeling, the network is able to fully implement domain-specific information extraction and cross-domain complementary information integration as well as maintaining the appropriate apparent intensity from a global perspective. In particular, we introduce the shifted windows mechanism into the self-attention and cross-attention, which allows our model to receive images with arbitrary sizes. On the other hand, the multi-scene image fusion problems are generalized to a unified framework with structure maintenance, detail preservation, and proper intensity control. Moreover, an elaborate loss function, consisting of SSIM loss, texture loss, and intensity loss, drives the network to preserve abundant texture details and structural information, as well as presenting optimal apparent intensity. Extensive experiments on both multi-modal image fusion and digital photography image fusion demonstrate the superiority of our SwinFusion compared to the state-of-the-art unified image fusion algorithms and task-specific alternatives. Implementation code and pre-trained weights can be accessed at https://github.com/Linfeng-Tang/SwinFusion.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/f31fcb340b2040d5169fedacabc23aecd99871f1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Hui Li
2. Xiaojun Wu
3. Josef Kittler
4. Yiannis Demiris
5. Xingchen Zhang
6. Linfeng Tang
7. Jun Huang
8. Yong Ma

In Computer Vision and Pattern Recognition

1. Hui Li
2. Luc Van Gool
3. Jiayi Ma
4. Radu Timofte
5. Xiaojun Wu
6. Josef Kittler
7. Yiannis Demiris
8. Yulun Zhang

In Media Technology

1. Hui Li
2. Yiannis Demiris
3. Xingchen Zhang
4. Radu Timofte
5. Luc Van Gool
6. Jiayi Ma
7. Xiaojun Wu
8. Josef Kittler

Citations in the corpus

• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, methodology): “, CGTF [29], SwinFusion [27], and YDTR [28]) first utilize CNN to extract shallow features and then employ Transformer to mine global interactions to facilitate adequate information integration.”
• Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “The existing methods can be applied to various image fusion problems, e.g., IFCNN [30], U2Fusion [2], and SwinFusion [31]; all of them are AL-based methods, intuitively reflecting the superiority of AL-based methods in image fusion.”
• Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “We selected five recent deep learning-based methods (IFCNN [23], SeAFusion [108], SwinFusion [121], U2Fusion [63], and YDTR [118]) that can represent the latest development in the VIF field and tested their performance on VIFB [167].”
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, methodology): “Many methods have been developed to tackle the MMIF challenges in recent years [35, 41, 44, 55, 73, 74, 82].”
• LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images (2023, methodology): “In this section, nine state-of-the-art fusion networks are chosen to preform a comparative evaluation: an auto-encoder based fusion network (DenseFuse) [22]; a GAN based fusion algorithm (FusionGAN) [32]; a novel end-to-end fusion method (IFCNN) [42]; an ISTA based fusion a network (CUNet) [31]; a n”
• A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, methodology): “As listed in Table I, the existing methods can be divided into two categories, namely Class 1 including TransFuser [56], SwinFusion [57], CMFNet [19], MFTransUNet [59], and Class 2 including EDFT [58].”