Diffusion Models in Vision: A Survey

By Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, Mubarak Shah. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

2,213 citations recorded by the source corpus.

Denoising diffusion models represent a recent emerging topic in computer vision, demonstrating remarkable results in the area of generative modeling. A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage. In the forward diffusion stage, the input data is gradually perturbed over several steps by adding Gaussian noise. In the reverse stage, a model is tasked at recovering the original input data by learning to gradually reverse the diffusion process, step by step. Diffusion models are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens, i.e., low speeds due to the high number of steps involved during sampling. In this survey, we provide a comprehensive review of articles on denoising diffusion models applied in vision, comprising both theoretical and practical contributions in the field. First, we identify and present three generic diffusion modeling frameworks, which are based on denoising diffusion probabilistic models, noise conditioned score networks, and stochastic differential equations. We further discuss the relations between diffusion models and other deep generative models, including variational auto-encoders, generative adversarial networks, energy-based models, autoregressive models and normalizing flows. Then, we introduce a multi-perspective categorization of diffusion models applied in computer vision. Finally, we illustrate the current limitations of diffusion models and envision some interesting directions for future research.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/efa1647594b236361610a20d507127f0586a379b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Statistical and Nonlinear Physics

Citations in the corpus

Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, background): “Recent compelling successes in 2D diffusion models [9, 22, 51] and large vision language models (e.g., CLIP model [49]) provide new possibilities for generating 3D assets using the strong priors of 2D diffusion models.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, methodology): “Generative models have been widely used in computer vision [56, 57, 58].”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, background): “A branch of related surveys [19], [20], [21], [22] reviews the progress of diffusion model in all fields, making them limited to providing limited coverage on the task of test-to-image synthesis.”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “Concept-based learning algorithms are a large and increasingly popular class of methods that can be used for both post-hoc ex-plainability and creating ante-hoc explainable models.”
A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal (2022, background): “However, most existing artificial intelligence (AI) models only imitate the learning procedure from experiences while ignoring the former [1], [2], [3], [4], thus making them less explainable and worse performances.”
Diffusion models in medical imaging: A comprehensive survey (2022, background): “There have also been review articles on diffusion models surfacing recently for computer vision tasks [20, 21, 22].”
Post-Training Quantization on Diffusion Models (2022, neutral): “Detailed information can be found in the surveys [43, 6].”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, positive): “model [38] and the ChatGPT based on transformer [10] has led to AIGC gaining a lot of attention.”
Diffusion Recommender Model (2023, background): “DMs recently have shown the capability of high-quality generation [4, 28].”
Probabilistic weather forecasting with machine learning (2024, neutral): “Diffusion models underpin many of the recent advances in modelling natural images, sounds and videos under the umbrella of generative AI 24,25 .”
Pix2Video: Video Editing using Image Diffusion (2023, neutral): “Diffusion-based algorithms [8, 18, 48] have emerged as the generative model of choice for image creation.”
DDGR: Continual Learning with Deep Diffusion-based Generative Replay (2023, methodology): “More details can be found in the work of Croitoru et al. (2022).”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, background): “Diffusion models [11, 28, 54] have shown impressive re-sults on 2D image generation.”
Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation (2023, background): “We refer to [4] for a thorough survey.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “Diffusion models has various applications [62], [199], [32], [38], [155], while the most notable is stable diffusion [153], which bridges the gap between text and image with its robust capability to generate coherent and contextually relevant images directly from textual descriptions.”