Deep learning for pixel-level image fusion: Recent advances and future prospects

By Yu Liu, Xun Chen, Zengfu Wang, Z. J. Wang, Rabab Kreidieh Ward, X. Wang. Information Fusion, 2018

655 citations recorded by the source corpus.

Abstract By integrating the information contained in multiple images of the same scene into one composite image, pixel-level image fusion is recognized as having high significance in a variety of fields including medical imaging, digital photography, remote sensing, video surveillance, etc. In recent years, deep learning (DL) has achieved great success in a number of computer vision and image processing problems. The application of DL techniques in the field of pixel-level image fusion has also emerged as an active topic in the last three years. This survey paper presents a systematic review of the DL-based pixel-level image fusion literature. Specifically, we first summarize the main difficulties that exist in conventional image fusion research and discuss the advantages that DL can offer to address each of these problems. Then, the recent achievements in DL-based image fusion are reviewed in detail. More than a dozen recently proposed image fusion methods based on DL techniques including convolutional neural networks (CNNs), convolutional sparse representation (CSR) and stacked autoencoders (SAEs) are introduced. At last, by summarizing the existing DL-based image fusion methods into several generic frameworks and presenting a potential DL-based framework for developing objective evaluation metrics, we put forward some prospects for the future study on this topic. The key issues and challenges that exist in each framework are discussed.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/2c10fa609a43cd1836a0a73c8231c3c2f40fb170

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Jiayi Ma
2. Hui Li
3. Xiaojun Wu
4. Josef Kittler
5. Yiannis Demiris
6. Xingchen Zhang
7. Han Xu
8. Junjun Jiang

In Media Technology

1. Jiayi Ma
2. Hui Li
3. Xiaojun Wu
4. Yiannis Demiris
5. Xingchen Zhang
6. Josef Kittler
7. Han Xu
8. Lorenzo Bruzzone

Citations in the corpus

• Deep learning in medical image registration: a survey (2019, background): “learning in medical image registration has not been extensively studied until the past three to four years. Though several review papers on deep learning in medical image analysis have been published [73,93,96,105,106,121,132,182], there are very few review papers that are specific to deep learning i”
• Deep learning in medical image registration: a review (2019, background): “Though several review papers on deep learning in medical image analysis have been published [73, 93, 96, 105, 106, 121, 132, 182], there are very few review papers that are specific to deep learning in medical image registration [60].”
• FusionGAN: A generative adversarial network for infrared and visible image fusion (2019, methodology): “In addition, the activity level measurement and fusion rule in most existing methods are designed in a manual way, and they have become more and more complex, having the limitations of implementation difficulty and computational cost [21].”
• Artificial intelligence in the creative industries: a review (2020, background): “A review of deep learning for pixel-level image fusion can be found in [288].”
• Visible and Infrared Image Fusion Using Deep Learning (2023, background): “In recent years, researchers have begun to perform VIF with deep learning techniques [20].”
• Multisource and Multitemporal Data Fusion in Remote Sensing: A Comprehensive Review of the State of the Art (2019, background): “There exists a wide range of approaches to data fusion; see, for example, [11]–[13].”
• Deep learning and medical image processing for coronavirus (COVID-19) pandemic: A survey (2020, neutral): “RNN can be used for datasets which involve time-series (Che, Purushotham, Cho, Sontag, & Liu, 2018), text, financial data, audio, video, etc. Generative Adversarial Networks (GANs) works on the concept of the generator network and the discriminator.”
• AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks (2021, methodology): “In contrast, visible image has high spatial resolution and contains abundant texture information [5].”
• Using labeled data to evaluate change detectors in a multivariate streaming environment (2009, neutral): “The others mostly adopt lightweight classifi-cation network as their backbone and carefully design the network to improve segmentation speed and remedy the accuracy drop.”