Learning a Deep Multi-Scale Feature Ensemble and an Edge-Attention Guidance for Image Fusion

By Jinyuan Liu, Xin Fan, Ji Jiang, Risheng Liu, Zhongxuan Luo. IEEE transactions on circuits and systems for video technology (Print), 2021

404 citations recorded by the source corpus.

Image fusion integrates a series of images acquired from different sensors, e.g., infrared and visible, outputting an image with richer information than either one. Traditional and recent deep-based methods have difficulties in preserving prominent structures and recovering vital textural details for practical applications. In this article, we propose a deep network for infrared and visible image fusion cascading a feature learning module with a fusion learning mechanism. Firstly, we apply a coarse-to-fine deep architecture to learn multi-scale features for multi-modal images, which enables discovering prominent common structures for later fusion operations. The proposed feature learning module requires no well-aligned image pairs for training. Compared with the existing learning-based methods, the proposed feature learning module can ensemble numerous examples from respective modals for training, increasing the ability of feature representation. Secondly, we design an edge-guided attention mechanism upon the multi-scale features to guide the fusion focusing on common structures, thus recovering details while attenuating noise. Moreover, we provide a new aligned infrared and visible image fusion dataset, RealStreet, collected in various practical scenarios for comprehensive evaluation. Extensive experiments on two benchmarks, TNO and RealStreet, demonstrate the superiority of the proposed method over the state-of-the-art in terms of both visual inspection and objective analysis on six evaluation metrics. We also conduct the experiments on the FLIR and NIR datasets, containing foggy weather and poor light conditions, to verify the generalization and robustness of the proposed method.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/a0b05ccd55e9fc0a44bf132be5ac20dd34607842

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation (2023, background): “Early efforts [8, 9, 10, 21, 22] tend to achieve excellent fusion effects by adjusting the network structure or loss functions.”
DATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer (2023, methodology): “, AUIF [1], DenseFuse [53], FusionGAN [6], GANMcC [55], RFN-Nest [54], CSF [56], and MFEIF [24], and two latest Transformer-based fusion models, i.”
CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, background): “The fusion targets include digital [45,84], multi-modal [36,70,88]”
YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer (2023, background): “Liu et al. [28] presented a deep network for infrared and visible image fusion, where a feature learning module with a fusion learning mechanism was cascaded.”
Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “In addition to the above methods, an interesting VIF method is the MFEIF proposed in [99].”
Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection (2022, methodology): “Early works plugged deep networks into the IVIF process acting as a module of feature extraction or weights generation [8,12, 13,16].”
SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, neutral): “, Autoencoder (AE)-based [16], [17], [18], Convolutional Neural Network (CNN)-based [19], [20], [21], [22], Generative Adversarial Network (GAN)-based [23], [24], [25], [26], and Transformer-based [27], [28], [29] methods.”