AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks

By Jing Li, Hongtao Huo, Chang Li, Renhua Wang, Qi Feng. IEEE transactions on multimedia, 2021

368 citations recorded by the source corpus.

Infrared and visible image fusion aims to describe the same scene from different aspects by combining complementary information of multi-modality images. The existing Generative adversarial networks (GAN) based infrared and visible image fusion methods cannot perceive the most discriminative regions, and hence fail to highlight the typical parts existing in infrared and visible images. To this end, we integrate multi-scale attention mechanism into both generator and discriminator of GAN to fuse infrared and visible images (AttentionFGAN). The multi-scale attention mechanism aims to not only capture comprehensive spatial information to help generator focus on the foreground target information of infrared image and background detail information of visible image, but also constrain the discriminators focus more on the attention regions rather than the whole input image. The generator of AttentionFGAN consists of two multi-scale attention networks and an image fusion network. Two multi-scale attention networks capture the attention maps of infrared and visible images respectively, so that the fusion network can reconstruct the fused image by paying more attention to the typical regions of source images. Besides, two discriminators are adopted to force the fused result keep more intensity and texture information from infrared and visible image respectively. Moreover, to keep more information of attention region from source images, an attention loss function is designed. Finally, the ablation experiments illustrate the effectiveness of the key parts of our method, and extensive qualitative and quantitative experiments on three public datasets demonstrate the advantages and effectiveness of AttentionFGAN compared with the other state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/6d96fe3d9f2d098d98de3222e50fe60def327bd4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yiannis Demiris
2. Xingchen Zhang
3. Linfeng Tang
4. Jiayi Ma
5. Yong Ma
6. Jun Huang
7. Xiaoguang Mei
8. Jinde Cao

In Media Technology

1. Yiannis Demiris
2. Xingchen Zhang
3. Jiayi Ma
4. Linfeng Tang
5. Yong Ma
6. Jun Huang
7. Jinde Cao
8. Shishuang Yu

Citations in the corpus

• Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “Also, the source visible image and infrared image are processed using two different encoders instead of being concatenated and processed by the same CNN. Li et al. [66] also proposed Atten-tionFGAN, which employs multiscale attention mechanism in both generator and discriminators.”
• YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer (2023, methodology): “Li et al. [2] developed FusionGAN by integrating a multi-scale attention mechanism into both generator and discriminator to perceive more discriminative regions, called At-tentionFGAN.”
• SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, background): “injected the multi-scale attention mechanism into the GAN-based fusion framework to encourage the generator and discriminator to pay more attention to the meaningful regions [58], [59].”
• Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “Unlike the CNN-based method, the generative adversarial network (GAN)-based method [14], [15], [16], [17] uses adversarial games between generators and discriminators to generate fused images containing high-contrast infrared images and visible image background textures.”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, background): “Thus, subsequent works, such as DDcGAN [50], AttentionFGAN [51], SDDGAN [52], and GAN-FM [53] design dual discriminators to maintain modal balance.”