NestFuse: An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models

By Hui Li, Xiaojun Wu, Tariq Salim Durrani. IEEE Transactions on Instrumentation and Measurement, 2020

716 citations recorded by the source corpus.

In this article, we propose a novel method for infrared and visible image fusion where we develop nest connection-based network and spatial/channel attention models. The nest connection-based network can preserve significant amounts of information from input data in a multiscale perspective. The approach comprises three key elements: encoder, fusion strategy, and decoder, respectively. In our proposed fusion strategy, spatial attention models and channel attention models are developed that describe the importance of each spatial position and of each channel with deep features. First, the source images are fed into the encoder to extract multiscale deep features. The novel fusion strategy is then developed to fuse these features for each scale. Finally, the fused image is reconstructed by the nest connection-based decoder. Experiments are performed on publicly available data sets. These exhibit that our proposed approach has better fusion performance than other state-of-the-art methods. This claim is justified through both subjective and objective evaluations. The code of our fusion method is available at https://github.com/hli1221/imagefusion-nestfuse.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/5a697dd36da51026a5bfbc7849a6ad4cbad8c764

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Radu Timofte
3. Jiayi Ma
4. Hui Li
5. Yulun Zhang
6. Kai Zhang
7. Yiannis Demiris
8. Xiaojun Wu

In Media Technology

1. Radu Timofte
2. Luc Van Gool
3. Hui Li
4. Jiayi Ma
5. Yiannis Demiris
6. Xingchen Zhang
7. Yulun Zhang
8. Kai Zhang

Citations in the corpus

• Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “Many AE-based methods [42], [51], [52], [55], [74], [106] only utilize RGB images to train the AE and directly apply the trained AE to infrared images.”
• RFN-Nest: An end-to-end residual fusion network for infrared and visible images (2021, methodology): ““Encoder & Decoder” denotes that the encoder and the decoder are fixed, the fusion strategy is “ add ”, “ max ”, “ l 1 -norm”, “ l ∗ -norm” or “SCA”.”
• STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection (2021, methodology): “Considering that a network without downsampling cannot extract multiscale features from source images, the nest connection-based networks are introduced in the NestFuse to extract information from source images in a multiscale perspective [23].”
• Infrared and Visible Image Fusion via Decoupling Network (2022, methodology): “Li et al. [6] introduced a Nest network [28] to construct an encoder from a multiscale perspective, which preserves a large amount of information about the source images.”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, background): “further introduced nest connection and attention mechanism to improve the fusion performance [31], [32].”
• SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, background): “introduced the dense connection [22] and nest connection [52], [53] to reinforce the feature extraction capability of the encoder.”
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, methodology): “For discriminative methods, auto encoder-based models [72, 23, 21, 28, 53, 22, 65] use encoders and decoders to extract features and fuse them on a high-dimensional manifold.”