Infrared and visible image fusion via detail preserving adversarial learning

By Jiayi Ma, Pengwei Liang, Wei Yu, Chen Chen, Xiaojie Guo, Jia Wu, Junjun Jiang. Information Fusion, 2020

420 citations recorded by the source corpus.

Abstract TargefTablets can be detected easily from the background of infrared images due to their significantly discriminative thermal radiations, while visible images contain textural details with high spatial resolution which are beneficial to the enhancement of target recognition. Therefore, fused images with abundant detail information and effective target areas are desirable. In this paper, we propose an end-to-end model for infrared and visible image fusion based on detail preserving adversarial learning. It is able to overcome the limitations of the manual and complicated design of activity-level measurement and fusion rules in traditional fusion methods. Considering the specific information of infrared and visible images, we design two loss functions including the detail loss and target edge-enhancement loss to improve the quality of detail information and sharpen the edge of infrared targets under the framework of generative adversarial network. Our approach enables the fused image to simultaneously retain the thermal radiation with sharpening infrared target boundaries in the infrared image and the abundant textural details in the visible image. Experiments conducted on publicly available datasets demonstrate the superiority of our strategy over the state-of-the-art methods in both objective metrics and visual impressions. In particular, our results look like enhanced infrared images with clearly highlighted and edge-sharpened targets as well as abundant detail information.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/56afe9ff014c7752f41664b6eaf0bcb323401233

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jiayi Ma
3. Radu Timofte
4. Hui Li
5. Ming-Hsuan Yang
6. Yulun Zhang
7. Tao A. Xiang
8. Chen Change Loy

In Media Technology

1. Jiayi Ma
2. Hui Li
3. Radu Timofte
4. Yiannis Demiris
5. Xingchen Zhang
6. Luc Van Gool
7. Linfeng Tang
8. Xiaojun Wu

Citations in the corpus

• LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images (2023, background): “works, including: FusionGAN [32] and FusionGANv2 [33], U2Fusion [27], CUNet [31], SwinFusion [34] and YDTR [35].”
• GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion (2021, methodology): “Subsequently, they introduced a detail loss and a target edge-enhancement loss based on FusionGAN to further enhance the texture details in the fused results [40].”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, background): “FusionGAN first models the image fusion task as the adversarial game between a generator and a discriminator, where the discrimi-nator forces the generator to retain more texture details from visible images [23], [49].”
• STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection (2021, methodology): “The GAN-based fusion methods establish an adversarial game between the fused image and source images, so as to make the fused image approximate the desired probability distribution without supervision [15], [16].”
• Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “In addition, various deep learning models, such as CNNs [29], [30], [31], GANs [6], [24], [32], [33] and transformers [34], [35], [36], [37], have been employed to perform VIF.”
• Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “Learning unsupervised region-wise correspondence Matching same or similar structure/content from two or more input data is a fundamental task in various downstream applications [47, 81] such as image stitching [10], image fusion [48], co-segmentation [23,76], image retrieval [88], object recognition”
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, methodology): “For generative methods [26, 23, 27], particularly the GAN family, adversarial training [7, 28, 30] is employed to generate fusion images following the same distribution as the source images.”
• SDNet: A Versatile Squeeze-and-Decomposition Network for Real-Time Image Fusion (2021, methodology): “In particular, Ma et al. (2020) proposed an unsupervised network to generate the decision map for fusion, which can indicate whether the pixel is focused.”
• DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion (2020, background): “In the light of Conditional GANs [ Mirza and Osindero, 2014 ] , detail preserving GAN [ Ma et al. , 2020 ] changes the loss function of FusionGAN for improving the quality of detail information and sharpening the target boundary.”
• U2Fusion: A Unified Unsupervised Image Fusion Network (2020, background): “Later, its variant [22] was proposed to sharpen the edges of thermal targets by introducing the target-enhancement loss.”
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, methodology): “In the era of DL, CNN-based models for MMIF can be categorized into four main classes: generative adversarial network (GAN)-based models [39,42,43], AE-based models [28, 29, 37, 82], unified models [26, 70, 71, 83, 85] and algorithm unrolling models [11,15,77,89].”
• DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion (2020, background): “(iv) In existing fusion methods based on traditional generative adversarial network (GAN) [21], [22], the fused image is trained to be similar to only one of the source images, leading to the loss of some information contained in the other source image.”
• A review of multimodal image matching: Methods and applications (2021, neutral): “improved in [480] by preserving better details.”
• Image Matching from Handcrafted to Deep Features: A Survey (2020, background): “Subsequently, they introduced a detail loss and a target edge-enhancement loss to further enrich the texture details (Ma et al. 2020).”
• RFN-Nest: An end-to-end residual fusion network for infrared and visible images (2021, methodology): “To eliminate the arbitrariness of handcrafted features and fusion strategies, several end-to-end fusion frameworks have been suggested [29] [30] [38] [39] [31] [40].”
• MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion (2018, methodology): “[31], [32], on the other hand, applied Genera-”