SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
By Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon. 2021
2,301 citations recorded by the source corpus.
Guided image synthesis enables everyday users to create and edit photo-realistic images with minimum effort. The key challenge is balancing faithfulness to the user input (e.g., hand-drawn colored strokes) and realism of the synthesized image. Existing GAN-based methods attempt to achieve such balance using either conditional GANs or GAN inversions, which are challenging and often require additional training data or loss functions for individual applications. To address these issues, we introduce a new image synthesis and editing method, Stochastic Differential Editing (SDEdit), based on a diffusion model generative prior, which synthesizes realistic images by iteratively denoising through a stochastic differential equation (SDE). Given an input image with user guide of any type, SDEdit first adds noise to the input, then subsequently denoises the resulting image through the SDE prior to increase its realism. SDEdit does not require task-specific training or inversions and can naturally achieve the balance between realism and faithfulness. SDEdit significantly outperforms state-of-the-art GAN-based methods by up to 98.09% on realism and 91.72% on overall satisfaction scores, according to a human perception study, on multiple tasks, including stroke-based image synthesis and editing as well as image compositing.
Subfields: Biophysics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Biophysics
- Richard Zhang
- Radu Timofte
- Ming-Hsuan Yang
- Alexei A. Efros
- Luc Van Gool
- Robin Rombach
- Antonio Torralba
- Chen Change Loy
In Computer Graphics and Computer-Aided Design
- Eli Shechtman
- Richard Zhang
- Alexei A. Efros
- Antonio Torralba
- Ziwei Liu
- Tali Dekel
- William T. Freeman
- Aleksander Holynski
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Chen Change Loy
- Andrea Vedaldi
In Generative Models
- Eli Shechtman
- Karsten Kreis
- Arash Vahdat
- Richard Zhang
- Jun-Yan Zhu
- Ming-Hsuan Yang
- Alexei A. Efros
- Luc Van Gool
Citations in the corpus
- Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “Numerous applications have emerged notably in the image domain, such as image manipulation, translation, super-resolution [8,32,35,43], in speech and text domains [5,6], or 3D point cloud [34].”
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “SDEdit [38] is a technique for editing images with a pretrained diffusion model, where a partially noised image is passed as input and denoised to produce a new edited image.”
- Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, positive): “We investigate the personalized text-to-audio generation by stochastic differential editing (Meng et al., 2021), which has been demonstrated to produce realistic samples with high-fidelity manipulation.”
- Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, methodology): “There exist prior works, which are similar in spirit to ours in their use of image generative models, including GANs [3,18,32,33,91] or diffusion models [13, 16, 28, 31, 55, 64–68, 72] to perform semantic segmentation [2,20,37,50,71,85].”
- Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation (2023, background): “Based on it, a multitude of image editing methods have emerged, including model fine-tuning for customized object generation [33], image-to-image translation [23], image inpainting [1], and object editing [12].”
- Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, critical): “However, injecting the low-resolution input via channel-wise concatenation, as per the first super-resolution module and SDx4 2 , is inadequate for addressing these artifacts.”
- MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, methodology): “Score-based diffusion models have been used for image editing [Meng et al., 2022, Saharia et al., 2021, Nichol et al., 2021] and our approach to video generation might be viewed as an analogy to classical image inpainting, but in the temporal dimension.”
- Improving Diffusion Model Efficiency Through Patching (2022, background): “Futhermore, pretrained diffusion models are well-suited for downstream tasks such as image inpainting [19] or stroke-to-image synthesis [20], often outperforming rival methods without any task-specific training.”
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, positive): “9, a 14-Method minute example video exhibits slight overexposure but re-tains overall high quality.”
- Latte: Latent Diffusion Transformer for Video Generation (2024, neutral): “…et al, 2020; Song et al, 2021b,a) are powerful deep generative models for various tasks in content creation, including image-to-image generation (Meng et al, 2022; Zhao et al, 2022; Saharia et al, 2022a; Parmar et al, 2023), text-to-image generation (Zhou et al, 2023; Rom-bach et al, 2022; Zhou…”
- An Edit Friendly DDPM Noise Space: Inversion and Manipulations (2023, background): “[16] who edit real images via user sketches or strokes (SDEdit).”
- Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “To guide the model toward generating images with a clean white backgrounds, inspired by SDEdit (Meng et al., 2022), we first create a white image of a 2x2 grid that has the same resolution as the output images, and initialize each sub-grid with a black 2D Gaussian blob that is placed at the center o”
- Consistency Models (2023, neutral): “Note that the above algorithm allows for many zero-shot data editing tasks such as inpainting, colorization, super-resolution, and SDEdit [MHS + 21].”
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (2023, background): “Apart from text-to-image generation, text-to-image diffusion models also can achieve text-guided image-to-image and inpainting with SDEdit [50].”