YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer

By Wei Tang, Fazhi He, Y. Liu. IEEE transactions on multimedia, 2023

421 citations recorded by the source corpus.

Infrared and visible image fusion is aims to generate a composite image that can simultaneously describe the salient target in the infrared image and texture details in the visible image of the same scene. Since deep learning (DL) exhibits great feature extraction ability in computer vision tasks, it has also been widely employed in handling infrared and visible image fusion issue. However, the existing DL-based methods generally extract complementary information from source images through convolutional operations, which results in limited preservation of global features. To this end, we propose a novel infrared and visible image fusion method, i.e., the Y-shape dynamic Transformer (YDTR). Specifically, a dynamic Transformer module (DTRM) is designed to acquire not only the local features but also the significant context information. Furthermore, the proposed network is devised in a Y-shape to comprehensively maintain the thermal radiation information from the infrared image and scene details from the visible image. Considering the specific information provided by the source images, we design a loss function that consists of two terms to improve fusion quality: a structural similarity (SSIM) term and a spatial frequency (SF) term. Extensive experiments on mainstream datasets illustrate that the proposed method outperforms both classical and state-of-the-art approaches in both qualitative and quantitative assessments. We further extend the YDTR to address other infrared and RGB-visible images and multi-focus images without fine-tuning, and the satisfactory fusion results demonstrate that the proposed method has good generalization capability.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/64de408ae19717febb8a6aaf0d6ff2d051d0ac77

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Hui Li
2. Xiaojun Wu
3. Jiayi Ma
4. Xiao-Jun Wu
5. Josef Kittler
6. Linfeng Tang
7. Yong Ma
8. Y. Liu

In Media Technology

1. Hui Li
2. Xiao-Jun Wu
3. Jiayi Ma
4. Xiaojun Wu
5. Josef Kittler
6. Linfeng Tang
7. Wei Tang
8. Y. Liu

Citations in the corpus

• DATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer (2023, methodology): “Recently, several Transformer-based infrared and visible image fusion models were proposed [65].”
• LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images (2023, methodology): “He, and Y. Liu, “YDTR: Infrared and visible image fusion via y-shape dynamic transformer,” IEEE Trans.”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, methodology): “, Autoencoder (AE)-based [16], [17], [18], Convolutional Neural Network (CNN)-based [19], [20], [21], [22], Generative Adversarial Network (GAN)-based [23], [24], [25], [26], and Transformer-based [27], [28], [29] methods.”
• CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach (2024, methodology): “Comparing with IFCNN [32], YDTR [34], U2Fusion [30] and DATFuse [42], the infrared objects are enhanced by the proposed CrossFuse (Fig.16, red box and yellow box).”