DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion

By Jiayi Ma, Han Xu, Junjun Jiang, Xiaoguang Mei, Xiao-Ping Zhang. IEEE Transactions on Image Processing, 2020

1,204 citations recorded by the source corpus.

In this paper, we proposed a new end-to-end model, termed as dual-discriminator conditional generative adversarial network (DDcGAN), for fusing infrared and visible images of different resolutions. Our method establishes an adversarial game between a generator and two discriminators. The generator aims to generate a real-like fused image based on a specifically designed content loss to fool the two discriminators, while the two discriminators aim to distinguish the structure differences between the fused image and two source images, respectively, in addition to the content loss. Consequently, the fused image is forced to simultaneously keep the thermal radiation in the infrared image and the texture details in the visible image. Moreover, to fuse source images of different resolutions, e.g., a low-resolution infrared image and a high-resolution visible image, our DDcGAN constrains the downsampled fused image to have similar property with the infrared image. This can avoid causing thermal radiation information blurring or visible texture detail loss, which typically happens in traditional methods. In addition, we also apply our DDcGAN to fusing multi-modality medical images of different resolutions, e.g., a low-resolution positron emission tomography image and a high-resolution magnetic resonance image. The qualitative and quantitative experiments on publicly available datasets demonstrate the superiority of our DDcGAN over the state-of-the-art, in terms of both visual effect and quantitative metrics. Our code is publicly available at https://github.com/jiayi-ma/DDcGAN.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/c65897d366831d95e5aaa3332b79c308554e5c49

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Jiayi Ma
2. Luc Van Gool
3. Hui Li
4. Radu Timofte
5. Yulun Zhang
6. Xiaojun Wu
7. Josef Kittler
8. Ming-Hsuan Yang

In Media Technology

1. Jiayi Ma
2. Hui Li
3. Radu Timofte
4. Luc Van Gool
5. Linfeng Tang
6. Yiannis Demiris
7. Xingchen Zhang
8. Yulun Zhang

Citations in the corpus

• SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, background): “…fusion and general image fusion can be further specified into four classes, including traditional framework [17]– [19], convolution neural network (CNN)-based framework [20], [21], auto-encoder (AE)-based framework [22], [23], and generative adversarial network (GAN)-based framework [24]– [26].”
• STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection (2021, background): “Ma et al. [11], for the first time, established an adversarial game between the fused result and the visible image to enhance the preservation of texture structures.”
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, methodology): “Among them, models based on Generative Adversarial Networks (GANs) [34, 35, 33, 26] are dominant.”
• GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion (2021, background): “In addition, they also introduced the dual-discriminator to better extract and reconstruct the information contained in source images [41].”
• RFN-Nest: An end-to-end residual fusion network for infrared and visible images (2021, methodology): “To eliminate the arbitrariness of handcrafted features and fusion strategies, several end-to-end fusion frameworks have been suggested [29] [30] [38] [39] [31] [40].”
• Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “Unlike the CNN-based method, the generative adversarial network (GAN)-based method [14], [15], [16], [17] uses adversarial games between generators and discriminators to generate fused images containing high-contrast infrared images and visible image background textures.”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, background): “Thus, subsequent works, such as DDcGAN [50], AttentionFGAN [51], SDDGAN [52], and GAN-FM [53] design dual discriminators to maintain modal balance.”
• U2Fusion: A Unified Unsupervised Image Fusion Network (2020, methodology): “We compare U2Fusion with five state-of-the-art methods: HMSD [42], GTF [19], DenseFuse [17], FusionGAN [5], and DDcGAN [24].”
• YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer (2023, methodology): “DL-based meth-ods mainly include convolutional neural network (CNN)-based methods [18], [20], [22], [23] and generative adversarial network (GAN)-based methods [2], [19], [21], [24].”
• Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection (2022, methodology): “Ma et al. apply an identical adversarial strategy to both the visible and infrared, which partially compensates the infrared information [21].”