RFN-Nest: An end-to-end residual fusion network for infrared and visible images

By Hui Li, Xiaojun Wu, Josef Kittler. Information Fusion, 2021

1,050 citations recorded by the source corpus.

Abstract In the image fusion field, the design of deep learning-based fusion methods is far from routine. It is invariably fusion-task specific and requires a careful consideration. The most difficult part of the design is to choose an appropriate strategy to generate the fused image for a specific task in hand. Thus, devising learnable fusion strategy is a very challenging problem in the community of image fusion. To address this problem, a novel end-to-end fusion network architecture (RFN-Nest) is developed for infrared and visible image fusion. We propose a residual fusion network (RFN) which is based on a residual architecture to replace the traditional fusion approach. A novel detail-preserving loss function, and a feature enhancing loss function are proposed to train RFN. The fusion model learning is accomplished by a novel two-stage training strategy. In the first stage, we train an auto-encoder based on an innovative nest connection (Nest) concept. Next, the RFN is trained using the proposed loss functions. The experimental results on public domain data sets show that, compared with the existing methods, our end-to-end fusion network delivers a better performance than the state-of-the-art methods in both subjective and objective evaluation. The code of our fusion method is available at https://github.com/hli1221/imagefusion-rfn-nest .

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/30d191b63102a138b10e05606688e30621d795b2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Radu Timofte
3. Hui Li
4. Jiayi Ma
5. Kai Zhang
6. Yulun Zhang
7. Yiannis Demiris
8. Xiaojun Wu

In Media Technology

1. Hui Li
2. Yiannis Demiris
3. Xingchen Zhang
4. Radu Timofte
5. Luc Van Gool
6. Jiayi Ma
7. Xiao-Jun Wu
8. Yulun Zhang

Citations in the corpus

• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, methodology): “Here, inspired by [28], we use a two-stage learning scheme to train our CDDFuse end-to-end. where L ir and L vis are the reconstruction losses for infrared and visible images, L decomp is the feature decomposition loss, and α 1 as well as α 2 are the tuning papameters.”
• YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer (2023, methodology): “Fromtoptobottom:infraredimage,visible image,thefusionresultsofRP[8],CVT[9],DenseFuse[18],FusionGAN[19],U2Fusion[20],GANMcC[24],RFN-Nest[23],CSF[29],PPTFusion[32],and theproposedYDTR.”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, methodology): “Subsequently, Li et al. further introduced nest connection and attention mechanism to improve the fusion performance [31], [32].”
• SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, background): “Afterwards, Li et al. introduced the dense connection [22] and nest connection [52], [53] to reinforce the feature extraction capability of the encoder.”
• CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach (2024, methodology): “Multi-stage image fusion methods divide the fusion process into several stages [20][28].”
• Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “The AE-based methods [4], [5], [6], [7], [8] use a pretraining approach, which usually trains the feature extraction capability of AE on natural datasets (e.g., MS-COCO [9]).”
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, methodology): “For discriminative methods, auto encoder-based models [72, 23, 21, 28, 53, 22, 65] use encoders and decoders to extract features and fuse them on a high-dimensional manifold.”
• Visible and Infrared Image Fusion Using Deep Learning (2023, background): “Another way is using visible-infrared image pairs to train a fusion module [73], [75].”
• LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images (2023, methodology): “The main reasons why “DFAT+RFN” does not achieve better performance is that the network structure is too simple to use with the decision-level based tracker, and the loss function of RFN also needs to be reconsidered.”
• DATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer (2023, methodology): “, AUIF [1], DenseFuse [53], FusionGAN [6], GANMcC [55], RFN-Nest [54], CSF [56], and MFEIF [24], and two latest Transformer-based fusion models, i.”
• Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection (2022, methodology): “We evaluate the fusion performance of TarDAL by making a comparison with 7 state-of-the-art methods, including DenseFuse [8], FusionGAN [22], RFN [9], GANMcC [23], DDcGAN [21], MFEIF [12], and U2Fusion [35].”