IFCNN: A general image fusion framework based on convolutional neural network

By Yu Zhang, Yu Liu, Peng Sun, Han Yan, Xiaolin Zhao, Li Zhang. Information Fusion, 2020

1,399 citations recorded by the source corpus.

Abstract In this paper, we propose a general image fusion framework based on the convolutional neural network, named as IFCNN. Inspired by the transform-domain image fusion algorithms, we firstly utilize two convolutional layers to extract the salient image features from multiple input images. Afterwards, the convolutional features of multiple input images are fused by an appropriate fusion rule (elementwise-max, elementwise-min or elementwise-mean), which is selected according to the type of input images. Finally, the fused features are reconstructed by two convolutional layers to produce the informative fusion image. The proposed model is fully convolutional, so it could be trained in the end-to-end manner without any post-processing procedures. In order to fully train the model, we have generated a large-scale multi-focus image dataset based on the large-scale RGB-D dataset (i.e., NYU-D2), which owns ground-truth fusion images and contains more diverse and larger images than the existing datasets for image fusion. Without finetuning on other types of image datasets, the experimental results show that the proposed model demonstrates better generalization ability than the existing image fusion models for fusing various types of images, such as multi-focus, infrared-visual, multi-modal medical and multi-exposure images. Moreover, the results also verify that our model has achieved comparable or even better results compared to the state-of-the-art image fusion algorithms on four types of image datasets.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/c59830c28faffa2a87d674e9f9f46470bf0c6cd0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jiayi Ma
3. Radu Timofte
4. Hui Li
5. Yulun Zhang
6. Xiaojun Wu
7. Josef Kittler
8. Ming-Hsuan Yang

In Media Technology

1. Jiayi Ma
2. Hui Li
3. Radu Timofte
4. Luc Van Gool
5. Linfeng Tang
6. Xiaojun Wu
7. Yulun Zhang
8. Josef Kittler

Citations in the corpus

• CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach (2024, methodology): “…fusion methods are chosen: a generative adversarial network (GAN) based fusion method (Fu-sionGAN) [21], a unified CNN based method (IFCNN) [32], a unified dense connection based fusion method (U2Fusion) [30], two transformer based fusion networks (YDTR [34], DAT-Fuse [42]), a joint…”
• LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images (2023, methodology): “Compared with the lightweight fusion network IFCNN [42] (0.”
• U2Fusion: A Unified Unsupervised Image Fusion Network (2020, background): “Then, it was extended to a general image fusion framework [26].”
• SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, methodology): “Nevertheless, all aforementioned methods adopt hand-crafted fusion strategies, e.g. , element-wise addition [13], element-wise weight summation [22], and element-wise maximum [20], to merge deep features, which obstruct the fusion models from achieving their optimal performance.”
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, methodology): “In the era of DL, CNN-based models for MMIF can be categorized into four main classes: generative adversarial network (GAN)-based models [39,42,43], AE-based models [28, 29, 37, 82], unified models [26, 70, 71, 83, 85] and algorithm unrolling models [11,15,77,89].”
• MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion (2018, methodology): “They include: cross bilateral filter method(CBF) [14], discrete co-sine harmonic wavelet transform method(DCHWT) [13], joint sparse representation-based method(JSR) [55], the JSR model with saliency detection method(JSRSD) [23], the gradient transfer fusion method(GTF) [30], weighted least square opt”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, background): “Similarly, researchers further developed a great number of general image fusion models ( e.g. , IFCNN [44], SGFusion [45]) to fully take advantage of their convenient potential.”
• NestFuse: An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models (2020, methodology): “We choose twelve typical and state-of-the-art fusion meth-ods to evaluate the fusion performance, including: cross bilateral filter fusion method (CBF) [39], discrete cosine harmonic wavelet transform fusion method (DCHWT) [40], joint SR based fusion method (JSR) [41], the joint sparse representation”
• STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection (2021, methodology): “Zhang et al. [12] utilized two convolutional layers to extract the salient features from source images.”
• LLVIP: A Visible-infrared Paired Dataset for Low-light Vision (2021, methodology): “In general, densefuse l1 and IFCNN perform best on the dataset, but they still have a lot of room for improvement.”
• Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “The existing methods can be applied to various image fusion problems, e.g., IFCNN [30], U2Fusion [2], and SwinFusion [31]; all of them are AL-based methods, intuitively reflecting the superiority of AL-based methods in image fusion.”
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, background): “Unified models [46, 52, 45, 54, 13] avoid the problem of lacking training data and ground truth for specific tasks.”
• FusionDN: A Unified Densely Connected Network for Image Fusion (2020, methodology): “These methods can be divided into spatial-domain meth-ods and transform-domain methods (Zhang et al. 2020).”