Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration

By Di Wang, Jinyuan Liu, Xin Fan, Risheng Liu. International Joint Conference on Artificial Intelligence, 2022

323 citations recorded by the source corpus.

Recent learning-based image fusion methods have marked numerous progress in pre-registered multi-modality data, but suffered serious ghosts dealing with misaligned multi-modality data, due to the spatial deformation and the difficulty narrowing cross-modality discrepancy. To overcome the obstacles, in this paper, we present a robust cross-modality generation-registration paradigm for unsupervised misaligned infrared and visible image fusion (IVIF). Specifically, we propose a Cross-modality Perceptual Style Transfer Network (CPSTN) to generate a pseudo infrared image taking a visible image as input. Benefiting from the favorable geometry preservation ability of the CPSTN, the generated pseudo infrared image embraces a sharp structure, which is more conducive to transforming cross-modality image alignment into mono-modality registration coupled with the structure-sensitive of the infrared image. In this case, we introduce a Multi-level Refinement Registration Network (MRRN) to predict the displacement vector field between distorted and pseudo infrared images and reconstruct registered infrared image under the mono-modality setting. Moreover, to better fuse the registered infrared images and visible images, we present a feature Interaction Fusion Module (IFM) to adaptively select more meaningful features for fusion in the Dual-path Interaction Fusion Network (DIFN). Extensive experimental results suggest that the proposed method performs superior capability on misaligned cross-modality image fusion.

Subfields: Biomedical Engineering, Media Technology, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/7726be6645e8864edbc1e1414369f60a7e99e145

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Yiannis Demiris
2. Xingchen Zhang
3. Linfeng Tang
4. Jun Huang
5. Yong Ma
6. Kai Zhang
7. Wei Zhong
8. Luc Van Gool

In Media Technology

1. Radu Timofte
2. Luc Van Gool
3. Kai Zhang
4. Yiannis Demiris
5. Xingchen Zhang
6. Yulun Zhang
7. Jiayi Ma
8. Linfeng Tang

In Radiology, Nuclear Medicine and Imaging

1. Jiayi Ma
2. Yiannis Demiris
3. Xingchen Zhang
4. Kai Zhang
5. Luc Van Gool
6. Deyu Meng

Citations in the corpus

• Visible and Infrared Image Fusion Using Deep Learning (2023, methodology): “For example, Wang et al. [111] first generates a pseudo-infrared image using a cross-modality perceptual style transfer network.”
• Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation (2023, methodology): “Multi-modality image fusion In recent years, deep learning based multi-modality image fusion approaches achieved significant progress[11, 12, 18, 19, 20].”
• SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness (2022, methodology): “For image fusion, we also select five state-of-the-art methods for comparison, including RFN-Nest [32], SeAFusion [22], UMF-CMGR [30], U2Fusion [43], and TarDAL [25].”
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, methodology): “Moreover, the pre-processing registration module [60, 14, 54, 61] can enhance the robustness for unregistered input images.”
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, unknown): “Additionally, adding a pre-processing registration module before the fusion module is proved to solve the misregistration of source images effectively [19,62,72].”