DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion

By Zixiang Zhao, Hao Bai, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, Luc Van Gool. IEEE International Conference on Computer Vision, 2023

404 citations recorded by the source corpus.

Multi-modality image fusion aims to combine different modalities to produce fused images that retain the complementary features of each modality, such as functional highlights and texture details. To leverage strong generative priors and address challenges such as unstable training and lack of interpretability for GAN-based generative methods, we propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM). The fusion task is formulated as a conditional generation problem under the DDPM sampling framework, which is further divided into an unconditional generation subproblem and a maximum likelihood subproblem. The latter is modeled in a hierarchical Bayesian manner with latent variables and inferred by the expectation-maximization (EM) algorithm. By integrating the inference solution into the diffusion sampling iteration, our method can generate high-quality fused images with natural image generative priors and cross-modality information from source images. Note that all we required is an unconditional pre-trained generative model, and no fine-tuning is needed. Our extensive experiments indicate that our approach yields promising fusion results in infrared-visible image fusion and medical image fusion. The code is available at https://github.com/Zhaozixiang1228/MMIF-DDFM.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/9b9e6306e82f2248eedb0527a0cbb6edbfaa51a8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiao-Jun Wu
2. Risheng Liu
3. Jinyuan Liu
4. Zhongxuan Luo
5. Wei Zhong
6. Guanyao Wu
7. Hui Li
8. Long Ma

In Media Technology

1. Xiao-Jun Wu
2. Jinyuan Liu
3. Risheng Liu
4. Zhongxuan Luo
5. Wei Zhong
6. Guanyao Wu
7. Hui Li
8. Long Ma

Citations in the corpus

• CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach (2024, methodology): “…connection based fusion method (U2Fusion) [30], two transformer based fusion networks (YDTR [34], DAT-Fuse [42]), a joint down-stream tasks (saliency object detection) fusion methods (IRFS) [35], a semantic based fsuio meth-ods (SemLA) [60], and a di ff usion model based fusion network (DDFM) [64].”
• Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation (2023, background): “Consequently, fusing the infrared and visible image [3, 4, 5, 6, 7] has become a mainstream solution for better scene understanding.”