U2Fusion: A Unified Unsupervised Image Fusion Network

By Han Xu, Jiayi Ma, Junjun Jiang, Xiaojie Guo, Haibin Ling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

1,963 citations recorded by the source corpus.

This study proposes a novel unified and unsupervised end-to-end image fusion network, termed as U2Fusion, which is capable of solving different fusion problems, including multi-modal, multi-exposure, and multi-focus cases. Using feature extraction and information measurement, U2Fusion automatically estimates the importance of corresponding source images and comes up with adaptive information preservation degrees. Hence, different fusion tasks are unified in the same framework. Based on the adaptive degrees, a network is trained to preserve the adaptive similarity between the fusion result and source images. Therefore, the stumbling blocks in applying deep learning for image fusion, e.g., the requirement of ground-truth and specifically designed metrics, are greatly mitigated. By avoiding the loss of previous fusion capabilities when training a single model for different tasks sequentially, we obtain a unified model that is applicable to multiple fusion tasks. Moreover, a new aligned infrared and visible image dataset, RoadScene (available at https://github.com/hanna-xu/RoadScene), is released to provide a new option for benchmark evaluation. Qualitative and quantitative experimental results on three typical image fusion tasks validate the effectiveness and universality of U2Fusion. Our code is publicly available at https://github.com/hanna-xu/U2Fusion.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/e4f5c628771ab05717a4025b07fc57f6e75d6d8c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jiayi Ma
3. Hui Li
4. Radu Timofte
5. Yiannis Demiris
6. Yulun Zhang
7. Xingchen Zhang
8. Kai Zhang

In Media Technology

1. Hui Li
2. Yiannis Demiris
3. Xingchen Zhang
4. Jiayi Ma
5. Radu Timofte
6. Luc Van Gool
7. Linfeng Tang
8. Yulun Zhang

Citations in the corpus

• SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, methodology): “7, GTF, SDNet, and U2Fusion cannot effectively present scene information in the visible images, since the lack of global information interaction and inappropriate intensity control.”
• Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection (2022, methodology): “We evaluate the fusion performance of TarDAL by making a comparison with 7 state-of-the-art methods, including DenseFuse [8], FusionGAN [22], RFN [9], GANMcC [23], DDcGAN [21], MFEIF [12], and U2Fusion [35].”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, methodology): “2) Experimental Configurations: Registration performance of our SuperFusion is compared with five state-of-the-art methods including DASC [62], RIFT [63], GLU-Net [76], UMF-CMGR [30], and CrossRAFT [66] on the MSRS [37] and RoadScene [43] datasets.”
• STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection (2021, methodology): “, DenseFuse [9], NestFuse [23], FusionGAN [15], GANMcC [24], IFCNN [12], PMGI [25], and U2Fusion [13].”
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, methodology): “Unified models [59, 66, 58, 68, 18] avoid the problem of lacking training data and ground truth for specific tasks.”
• CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach (2024, methodology): “Comparing with IFCNN [32], YDTR [34], U2Fusion [30] and DATFuse [42], the infrared objects are enhanced by the proposed CrossFuse (Fig.16, red box and yellow box).”
• LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images (2023, methodology): “In recent years, several multi-modal datasets have been released: KAIST [25], RGBT-234 [26], and RoadScene [27].”
• Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “It can [5], FusionGAN [15], U2Fusion [2], STD [10], SeAFusion [3] method, our method without DDBs, and our method, respectively.”
• GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion (2021, methodology): “At present, there are also some deep learning-based methods that can uniformly realize the abovementioned multiple image fusion tasks and can produce promising results, such as U2Fusion [38] and PMGI [39].”
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, methodology): “In this section, we test CDDFuse on the three test sets and compare the fusion results with the state-of-the-art meth-ods including DIDFuse [88], U2Fusion [70], SDNet [82], RFNet [72], TarDAL [35], DeFusion [32] and ReCoNet [19].”
• YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer (2023, methodology): “Fromtoptobottom:infraredimage,visible image,thefusionresultsofRP[8],CVT[9],DenseFuse[18],FusionGAN[19],U2Fusion[20],GANMcC[24],RFN-Nest[23],CSF[29],PPTFusion[32],and theproposedYDTR.”
• DATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer (2023, background): “On the one hand, conventional fusion methods generally need to design fusion strategies manually, which makes it increasingly complex to pursue better fusion results [28].”