Infrared and visible image fusion methods and applications: A survey

By Jiayi Ma, Yong Ma, Chang Li. Information Fusion, 2018

1,572 citations recorded by the source corpus.

Abstract Infrared images can distinguish targets from their backgrounds based on the radiation difference, which works well in all-weather and all-day/night conditions. By contrast, visible images can provide texture details with high spatial resolution and definition in a manner consistent with the human visual system. Therefore, it is desirable to fuse these two types of images, which can combine the advantages of thermal radiation information in infrared images and detailed texture information in visible images. In this work, we comprehensively survey the existing methods and applications for the fusion of infrared and visible images. First, infrared and visible image fusion methods are reviewed in detail. Meanwhile, image registration, as a prerequisite of image fusion, is briefly introduced. Second, we provide an overview of the main applications of infrared and visible image fusion. Third, the evaluation metrics of fusion performance are discussed and summarized. Fourth, we select eighteen representative methods and nine assessment metrics to conduct qualitative and quantitative experiments, which can provide an objective performance reference for different fusion methods and thus support relative engineering with credible and solid evidence. Finally, we conclude with the current status of infrared and visible image fusion and deliver insightful discussions and prospects for future work. This survey can serve as a reference for researchers in infrared and visible image fusion and related fields.

Subfields: Aerospace Engineering, Media Technology, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/27adceb33a98e55f263a495dbe66baf22b5cf789

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Jiayi Ma
2. Junjun Jiang
3. Xiaojie Guo
4. Han Xu
5. Xingyu Jiang
6. Y. Liu
7. Fazhi He
8. Wei Tang

In Media Technology

1. Jiayi Ma
2. Hui Li
3. Yiannis Demiris
4. Xingchen Zhang
5. Linfeng Tang
6. Xiaojun Wu
7. Josef Kittler
8. Radu Timofte

In Radiology, Nuclear Medicine and Imaging

1. Han Xu
2. Jiayi Ma
3. Xiaojie Guo
4. Junjun Jiang
5. Y. Liu
6. Fazhi He
7. Wei Tang
8. Yang Xiao

Citations in the corpus

• Visible and Infrared Image Fusion Using Deep Learning (2023, background): “Conventional VIF methods can be generally classified into several types according to their corresponding theories, namely multiscale transformbased methods, sparse representation-based methods, subspacebased methods, saliency-based methods, hybrid methods, and other methods [19].”
• LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images (2023, background): “The aim of this task is to synthesize a single image, which contains the complementary information from multiple input images [1][2].”
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, neutral): “The detail of metrics is in [32].”
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, background): “The details of these metrics can be found in [40].”
• FusionDN: A Unified Densely Connected Network for Image Fusion (2020, methodology): “Also in infrared and visible image fusion, Fu-sionGAN (Ma et al. 2019) and its variants (Ma et al. 2020; Xu et al. 2019) established an adversarial game between a generator and a discriminator.”
• DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion (2020, background): “Therefore, their fusion plays an important role in military and civilian applications [6], [7].”
• DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion (2020, background): “More details for these metrics can be seen in [Ma et al., 2019a].”
• Artificial intelligence in the creative industries: a review (2020, background): “Ma [193] shows that a guided filtering-based fusion [194] achieves the best results based on the visual information fidelity (VIF) metric, but proposed that fused images with very low correlation coefficients (CC), measuring the degree of linear correlation between the fused image its source images,”
• Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection (2018, methodology): “In this paper we focus on network based multispectral fusion, but we should note that other methods exist than network based ones (see [36, 37] for surveys of visible and infrared image fusion).”
• GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion (2021, background): “We denote these three losses as L D vis , L D ir , and L D fused , that is Considering the 1 × 2 vectors output by the discriminator, we have P vis = D ( x ) [ 1 ] and P ir = D ( x ) [ 2 ] .”
• FusionGAN: A generative adversarial network for infrared and visible image fusion (2019, background): “In recent years, many studies have proposed various fusion metrics, but it seems that none of them is definitely better than the others [41].”
• STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection (2021, background): “Therefore, the fused images obtained by multiscale transform can have a pleasing visual effect [1].”
• A review of multimodal image matching: Methods and applications (2021, background): “Formally, image fusion extracts the most meaningful information from images acquired by different sensors or under different shooting settings, and combines the information to generate a single image, which contains more abundant information and is more conducive to subsequent applications [13].”
• YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer (2023, methodology): “We replace DTRM i ( i ∈ [1 , 3] ) with an encoder to investigate the influence of removing one DTRM on the fusion performance.”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, background): “…researchers to integrate useful information from multi-modal images into a single fused image to facilitate human perception and subsequent visual applications, such as object detection, object tracking, semantic segmentation, security surveillance, and military reconnaissance [2], [6], [7].”