GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion

By Jiayi Ma, Hao Zhang, Zhenfeng Shao, Pengwei Liang, Han Xu. IEEE Transactions on Instrumentation and Measurement, 2021

673 citations recorded by the source corpus.

Visible images contain rich texture information, whereas infrared images have significant contrast. It is advantageous to combine these two kinds of information into a single image so that it not only has good contrast but also contains rich texture details. In general, previous fusion methods cannot achieve this goal well, where the fused results are inclined to either a visible or an infrared image. To address this challenge, a new fusion framework called generative adversarial network with multiclassification constraints (GANMcC) is proposed, which transforms image fusion into a multidistribution simultaneous estimation problem to fuse infrared and visible images in a more reasonable way. We adopt a generative adversarial network with multiclassification to estimate the distributions of visible light and infrared domains at the same time, in which the game of multiclassification discrimination will make the fused result to have these two distributions in a more balanced manner, so as to have significant contrast and rich texture details. In addition, we design a specific content loss to constrain the generator, which introduces the idea of main and auxiliary into the extraction of gradient and intensity information, which will enable the generator to extract more sufficient information from source images in a complementary manner. Extensive experiments demonstrate the advantages of our GANMcC over the state-of-the-art methods in terms of both qualitative effect and quantitative metric. Moreover, our method can achieve good fused results even the visible image is overexposed. Our code is publicly available at https://github.com/jiayi-ma/GANMcC.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/ed0959d722f06e2fb2862601aceae3ec1a511db9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, methodology): “For generative methods [34, 31, 35], particularly the GAN family, adversarial training [7, 36, 38] is employed to generate fusion images following the same distribution as the source images.”
SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, neutral): “…deep learning-based frameworks can be classiﬁed into four categories, i.e. , Autoencoder (AE)-based [16], [17], [18], Convolutional Neural Network (CNN)-based [19], [20], [21], [22], Generative Adversarial Network (GAN)-based [23], [24], [25], [26], and Transformer-based [27], [28], [29] methods.”
DATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer (2023, methodology): “, AUIF [1], DenseFuse [53], FusionGAN [6], GANMcC [55], RFN-Nest [54], CSF [56], and MFEIF [24], and two latest Transformer-based fusion models, i.”
Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “[101] proposed to use a multiclassification-based discriminator to achieve a balance between the visible distribution and infrared distribution.”
Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection (2022, methodology): “We evaluate the fusion performance of TarDAL by making a comparison with 7 state-of-the-art methods, including DenseFuse [8], FusionGAN [22], RFN [9], GANMcC [23], DDcGAN [21], MFEIF [12], and U2Fusion [35].”
YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer (2023, methodology): “Fromtoptobottom:infraredimage,visible image,thefusionresultsofRP[8],CVT[9],DenseFuse[18],FusionGAN[19],U2Fusion[20],GANMcC[24],RFN-Nest[23],CSF[29],PPTFusion[32],and theproposedYDTR.”
SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, background): “ANVF [3], DenseFuse [22], and GANMcC [82] are the task-speciﬁc comparison algorithms for VIS-NIR.”
STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection (2021, methodology): “, DenseFuse [9], NestFuse [23], FusionGAN [15], GANMcC [24], IFCNN [12], PMGI [25], and U2Fusion [13].”
CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, methodology): “In the era of DL, CNN-based models for MMIF can be categorized into four main classes: generative adversarial network (GAN)-based models [39, 42, 43], AE-based models [28, 29, 37, 82], unified models [26, 70, 71, 83, 85] and algorithm unrolling models [11, 15, 77, 89].”
Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “In addition, further achieving the adversarial balance will result in that the luminance information of the images cannot be wholly retained in the fused image [14], [15].”