STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection

By Jiayi Ma, Linfeng Tang, Meilong Xu, Hao Zhang, Guobao Xiao. IEEE Transactions on Instrumentation and Measurement, 2021

509 citations recorded by the source corpus.

In this article, we propose an infrared and visible image fusion network based on the salient target detection, termed STDFusionNet, which can preserve the thermal targets in infrared images and the texture structures in visible images. First, a salient target mask is dedicated to annotating regions of the infrared image that humans or machines pay more attention to, so as to provide spatial guidance for the integration of different information. Second, we combine this salient target mask to design a specific loss function to guide the extraction and reconstruction of features. Specifically, the feature extraction network can selectively extract salient target features from infrared images and background texture features from visible images, while the feature reconstruction network can effectively fuse these features and reconstruct the desired results. It is worth noting that the salient target mask is only required in the training phase, which enables the proposed STDFusionNet to be an end-to-end model. In other words, our STDFusionNet can fulfill salient target detection and key information fusion in an implicit manner. Extensive qualitative and quantitative experiments demonstrate the superiority of our fusion algorithm over the current state of the arts, where our algorithm is much faster and the fusion results look like high-quality visible images with clear highlighted infrared targets. Moreover, the experimental results on the public datasets reveal that our algorithm can improve the entropy (EN), mutual information (MI), visual information fidelity (VIF), and spatial frequency (SF) metrics with about 1.25%, 22.65%, 4.3%, and 0.89% gains, respectively. Our code is publicly available at https://github.com/jiayi-ma/STDFusionNet.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/d0afb32c90518b5906268b2db74ddb18538f1d6a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yiannis Demiris
2. Xingchen Zhang
3. Linfeng Tang
4. Jiayi Ma
5. Yong Ma
6. Jun Huang
7. Xiaoguang Mei
8. Jinde Cao

In Media Technology

1. Yiannis Demiris
2. Xingchen Zhang
3. Jiayi Ma
4. Linfeng Tang
5. Yong Ma
6. Jun Huang
7. Jinde Cao
8. Shishuang Yu

Citations in the corpus

• Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “[31] defined the VIF process as the fusion of texture information in the visible image and salient target in the infrared image.”
• SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, methodology): “Another type of CNN-based image fusion framework is utilizing CNN to learn a direct mapping between source images and fused images (or focus map) in an end-to-end manner [2], [44].”
• Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “It can [5], FusionGAN [15], U2Fusion [2], STD [10], SeAFusion [3] method, our method without DDBs, and our method, respectively.”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, unknown): “, Autoencoder (AE)-based [16], [17], [18], Convolutional Neural Network (CNN)-based [19], [20], [21], [22], Generative Adversarial Network (GAN)-based [23], [24], [25], [26], and Transformer-based [27], [28], [29] methods.”