Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review

By Qiang Zhang, Yi Liu, Rick S. Blum, Jungong Han, Dacheng Tao. Information Fusion, 2018

446 citations recorded by the source corpus.

Abstract As a result of several successful applications in computer vision and image processing, sparse representation (SR) has attracted significant attention in multi-sensor image fusion. Unlike the traditional multiscale transforms (MSTs) that presume the basis functions, SR learns an over-complete dictionary from a set of training images for image fusion, and it achieves more stable and meaningful representations of the source images. By doing so, the SR-based fusion methods generally outperform the traditional MST image fusion methods in both subjective and objective tests. In addition, they are less susceptible to mis-registration among the source images, thus facilitating the practical applications. This survey paper proposes a systematic review of the SR-based multi-sensor image fusion literature, highlighting the pros and cons of each category of approaches. Specifically, we start by performing a theoretical investigation of the entire system from three key algorithmic aspects, (1) sparse representation models; (2) dictionary learning methods; and (3) activity levels and fusion rules. Subsequently, we show how the existing works address these scientific problems and design the appropriate fusion rules for each application such as multi-focus image fusion and multi-modality (e.g., infrared and visible) image fusion. At last, we carry out some experiments to evaluate the impact of these three algorithmic components on the fusion performance when dealing with different applications. This article is expected to serve as a tutorial and source of reference for researchers preparing to enter the field or who desire to employ the sparse representation theory in other fields.

Subfields: Biomedical Engineering, Media Technology

View Paper: https://www.semanticscholar.org/paper/440a2960d63b74d1589257605b91a7dde767f91d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Jiayi Ma
2. Hao Zhang
3. Wei Zhong
4. Karthik Ramani
5. Y. Liu
6. Junjun Jiang
7. Haibin Ling
8. Han Xu

In Media Technology

1. Jiayi Ma
2. Hao Zhang
3. Yiannis Demiris
4. Xingchen Zhang
5. Hui Li
6. Wei Tang
7. Y. Liu
8. Fazhi He

Citations in the corpus

• GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion (2021, methodology): “Typical traditional methods are sparse representation-based methods [7], [8], multiscale transform-based methods [9], [10], subspace-based methods [11], saliency-based method [12], [13], and hybrid methods [14], [15].”
• U2Fusion: A Unified Unsupervised Image Fusion Network (2020, background): “; ii) sparse representation [14]; iii) subspace analysis, e.g. , independent component analysis (ICA), principal component analysis (PCA), non-negative matrix factorization (NMF), etc .”
• AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks (2021, methodology): “Sparse representation (SR) has also been used to infrared and visible image fusion, because the SR based methods can enhance the representation of the source images by learning an over-complete dictionary from some high-quality images [14]–[16].”
• Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection (2022, background): “Traditional multi-scale transform [10, 24], optimization model [16, 20, 41], spare representation [37, 43], and subspace methods attempt to discover intrinsic common features of the two modalities and to design appropriate weighting rules for fusion.”
• First-Person View Hand Segmentation of Multi-Modal Hand Activity Video Dataset (2020, methodology): “Multiple modalities are used to provide distinctive features to the networks [14, 56, 60] and has been an emerging area [11, 26, 44] due to the enhancement of computation power and sensors.”
• FusionDN: A Unified Densely Connected Network for Image Fusion (2020, background): “…to other domains, and the fusion process is completed in these transformed domains, including multi-scale transform (e.g., pyramid, wavelet, shearlet, discrete coscine transform), sparse representation (Zhang et al. 2018), hybrid (Paramanandham and Rajendiran 2018), subspace and other methods.”
• YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer (2023, methodology): “The conventional methods primarily involve multi-scale transform (MST)-based methods [8], [9], [11], sparse representation (SR)-based methods [12], [15], total variation (TV)-based meth-ods [10], [14], hybrid methods [13], [16], and other methods [1], [17].”
• DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion (2020, background): “…can be divided into different categories, including multi-scale transform-based methods [9], [10], sparse representation-based methods [11], [12], neural network-based methods [13], subspace-based methods [14], saliency-based methods [15], hybrid meth-ods [16], and other fusion methods…”