Rethinking the Image Fusion: A Fast Unified Image Fusion Network based on Proportional Maintenance of Gradient and Intensity

By Han Xu, Yang Xiao, Xiaojie Guo, Jiayi Ma. AAAI Conference on Artificial Intelligence, 2020

610 citations recorded by the source corpus.

In this paper, we propose a fast unified image fusion network based on proportional maintenance of gradient and intensity (PMGI), which can end-to-end realize a variety of image fusion tasks, including infrared and visible image fusion, multi-exposure image fusion, medical image fusion, multi-focus image fusion and pan-sharpening. We unify the image fusion problem into the texture and intensity proportional maintenance problem of the source images. On the one hand, the network is divided into gradient path and intensity path for information extraction. We perform feature reuse in the same path to avoid loss of information due to convolution. At the same time, we introduce the pathwise transfer block to exchange information between different paths, which can not only pre-fuse the gradient information and intensity information, but also enhance the information to be processed later. On the other hand, we define a uniform form of loss function based on these two kinds of information, which can adapt to different fusion tasks. Experiments on publicly available datasets demonstrate the superiority of our PMGI over the state-of-the-art in terms of both visual effect and quantitative metric in a variety of fusion tasks. In addition, our method is faster compared with the state-of-the-art.

Subfields: Biomedical Engineering, Media Technology

View Paper: https://www.semanticscholar.org/paper/f13c82cc881f2438881994e5f359f8f00420be0f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Hui Li
2. Xiaojun Wu
3. Josef Kittler
4. Yiannis Demiris
5. Xingchen Zhang
6. Linfeng Tang
7. Hao Zhang
8. Jiayi Ma

In Media Technology

1. Jiayi Ma
2. Hui Li
3. Yiannis Demiris
4. Xingchen Zhang
5. Linfeng Tang
6. Radu Timofte
7. Luc Van Gool
8. Xiaojun Wu

Citations in the corpus

• GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion (2021, background): “At present, there are also some deep learning-based methods that can uniformly realize the abovementioned multiple image fusion tasks and can produce promising results, such as U2Fusion [38] and PMGI [39].”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, background): “They first manually assigned appropriate weights for sub-losses according to the characteristics of different fusion missions [40], and further developed an adaptive decision block to dynamically assign weights for sub-loss terms to cope with the complex scenario [41].”
• SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, methodology): “The four unified image fusion algorithms are IFCNN [20], PMGI [16], SDNet [60], and U2Fusion [10].”
• SDNet: A Versatile Squeeze-and-Decomposition Network for Real-Time Image Fusion (2021, methodology): “Our preliminary version PMGI (Zhang et al. 2020) proposed a new image fusion network based on proportional maintenance of gradient and intensity information, which can realize a variety of image fusion tasks.”
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, methodology): “Through cross-task learning, the unified models can alleviate the problems of limited training data and missing ground-truth [83].”
• RFN-Nest: An end-to-end residual fusion network for infrared and visible images (2021, methodology): “To eliminate the arbitrariness of handcrafted features and fusion strategies, several end-to-end fusion frameworks have been suggested [29] [30] [38] [39] [31] [40].”
• Infrared and Visible Image Fusion via Decoupling Network (2022, background): “…autonomously based on the gap between the output and the final target (i.e., the model autonomously learns all the steps between the initial input stage and the final output result), which allows the trained model to generate fused images directly from the source images [2], [10], [11], [12], [13].”
• Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “General image fusion methods also contain CNNbased methods [23], [61], [62], [63], AE-based methods [82],”