CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion

By Zixiang Zhao, Hao Bai, Jiangshe Zhang, Yulun Zhang, Shuang Xu, Zudi Lin, Radu Timofte, Luc Van Gool. Computer Vision and Pattern Recognition, 2022

844 citations recorded by the source corpus.

Multi-modality (MM) image fusion aims to render fused images that maintain the merits of different modalities, e.g., functional highlight and detailed textures. To tackle the challenge in modeling cross-modality features and decomposing desirable modality-specific and modality-shared features, we propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. Firstly, CDDFuse uses Restormer blocks to extract cross-modality shallow features. We then introduce a dual-branch Transformer-CNN feature extractor with Lite Transformer (LT) blocks leveraging long-range attention to handle low-frequency global features and Invertible Neural Networks (INN) blocks focusing on extracting high-frequency local information. A correlation-driven loss is further proposed to make the low-frequency features correlated while the high-frequency features uncorrelated based on the embedded information. Then, the LT-based global fusion and INN-based local fusion layers output the fused image. Extensive experiments demonstrate that our CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. We also show that CDDFuse can boost the performance in downstream infrared-visible semantic segmentation and object detection in a unified benchmark. The code is available at https://github.om/haozixiang1228/MMIF-CDDFuse.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/d80734254cd020d40170fe0d438e457fe2ded808

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation (2023, background): “Consequently, fusing the infrared and visible image [3, 4, 5, 6, 7] has become a mainstream solution for better scene understanding.”
DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, methodology): “We first give a commonly-used loss function [17, 51, 22, 55] for the image fusion task:”
CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach (2024, background): “…multimodal image fusion also has numerous applications in other fields such as medical diagnosis [12][13], surveillance [14][15], remote sensing [16][17], and robotics [18][19] etc. Deep learning, as a widely popular technology, is no exception in multimodal image fusion field [20][21][22][23][24].”