FusionGAN: A generative adversarial network for infrared and visible image fusion

By Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, Junjun Jiang. Information Fusion, 2019

2,045 citations recorded by the source corpus.

Abstract Infrared images can distinguish targets from their backgrounds on the basis of difference in thermal radiation, which works well at all day/night time and under all weather conditions. By contrast, visible images can provide texture details with high spatial resolution and definition in a manner consistent with the human visual system. This paper proposes a novel method to fuse these two types of information using a generative adversarial network, termed as FusionGAN. Our method establishes an adversarial game between a generator and a discriminator, where the generator aims to generate a fused image with major infrared intensities together with additional visible gradients, and the discriminator aims to force the fused image to have more details existing in visible images. This enables that the final fused image simultaneously keeps the thermal radiation in an infrared image and the textures in a visible image. In addition, our FusionGAN is an end-to-end model, avoiding manually designing complicated activity level measurements and fusion rules as in traditional methods. Experiments on public datasets demonstrate the superiority of our strategy over state-of-the-arts, where our results look like sharpened infrared images with clear highlighted targets and abundant details. Moreover, we also generalize our FusionGAN to fuse images with different resolutions, say a low-resolution infrared image and a high-resolution visible image. Extensive results demonstrate that our strategy can generate clear and clean fused images which do not suffer from noise caused by upsampling of infrared information.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/2fb54f07b3322a6a9971f7e6a2504edecc124fee

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Jiayi Ma
2. Hui Li
3. Luc Van Gool
4. Yiannis Demiris
5. Radu Timofte
6. Xingchen Zhang
7. Xiaojun Wu
8. Josef Kittler

In Media Technology

1. Hui Li
2. Jiayi Ma
3. Yiannis Demiris
4. Xingchen Zhang
5. Linfeng Tang
6. Radu Timofte
7. Xiaojun Wu
8. Xiao-Jun Wu

Citations in the corpus

• AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks (2021, methodology): “Ma et al. first introduced the GAN into infrared and visible image fusion task, termed as FusionGAN [25].”
• Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “It can [5], FusionGAN [15], U2Fusion [2], STD [10], SeAFusion [3] method, our method without DDBs, and our method, respectively.”
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, methodology): “Third, the forward propagation of fusion networks often causes the loss of high-frequency information [42,89].”
• A review of multimodal image matching: Methods and applications (2021, neutral): “[479] proposed an end-to-end model called Fusion-”
• DATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer (2023, methodology): “Specifically, 361 pairs of infrared and RGB-visible images are downloaded from the MSRS dataset [80] 29], AUIF [1], DenseFuse [53], FusionGAN [6], GANMcC [55], RFN-Nest [54], CSF [56], MFEIF [24], PPTFusion [67], SwinFuse [7], and the proposed method.”
• YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer (2023, methodology): “Fromtoptobottom:infraredimage,visible image,thefusionresultsofRP[8],CVT[9],DenseFuse[18],FusionGAN[19],U2Fusion[20],GANMcC[24],RFN-Nest[23],CSF[29],PPTFusion[32],and theproposedYDTR.”
• Using labeled data to evaluate change detectors in a multivariate streaming environment (2009, positive): “FastSCNN [18] adopts the skip connection in the deep convolutional neural network and proposes a shallow learning module to downsam-ple for fast and efficient multi-branch low-level feature extraction.”
• FusionDN: A Unified Densely Connected Network for Image Fusion (2020, methodology): “Also in infrared and visible image fusion, Fu-sionGAN (Ma et al. 2019) and its variants (Ma et al. 2020; Xu et al. 2019) established an adversarial game between a generator and a discriminator.”
• U2Fusion: A Unified Unsupervised Image Fusion Network (2020, methodology): “We compare U2Fusion with five state-of-the-art methods: HMSD [42], GTF [19], DenseFuse [17], FusionGAN [5], and DDcGAN [24].”
• MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion (2018, methodology): “They include: cross bilateral filter method(CBF) [14], discrete cosine harmonic wavelet transform method(DCHWT) [13], joint sparse representation-based method(JSR) [55], the JSR model with saliency detection method(JSRSD) [23], the gradient transfer fusion method(GTF) [30], weighted least square opt”
• LLVIP: A Visible-infrared Paired Dataset for Low-light Vision (2021, methodology): “Deep learning algorithms [12, 10, 23] have achieved great success in the image fusion field.”
• Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration (2022, neutral): “, 2020], FGAN [Ma et al., 2019], GAN-FM [Zhang et al.”
• Artificial intelligence in the creative industries: a review (2020, neutral): “in [191]), with an example of image blending using a guided mask in [192].”