SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations

By Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon. 2021

2,301 citations recorded by the source corpus.

Guided image synthesis enables everyday users to create and edit photo-realistic images with minimum effort. The key challenge is balancing faithfulness to the user input (e.g., hand-drawn colored strokes) and realism of the synthesized image. Existing GAN-based methods attempt to achieve such balance using either conditional GANs or GAN inversions, which are challenging and often require additional training data or loss functions for individual applications. To address these issues, we introduce a new image synthesis and editing method, Stochastic Differential Editing (SDEdit), based on a diffusion model generative prior, which synthesizes realistic images by iteratively denoising through a stochastic differential equation (SDE). Given an input image with user guide of any type, SDEdit first adds noise to the input, then subsequently denoises the resulting image through the SDE prior to increase its realism. SDEdit does not require task-specific training or inversions and can naturally achieve the balance between realism and faithfulness. SDEdit significantly outperforms state-of-the-art GAN-based methods by up to 98.09% on realism and 91.72% on overall satisfaction scores, according to a human perception study, on multiple tasks, including stroke-based image synthesis and editing as well as image compositing.

Subfields: Biophysics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

  1. Richard Zhang
  2. Radu Timofte
  3. Ming-Hsuan Yang
  4. Alexei A. Efros
  5. Luc Van Gool
  6. Robin Rombach
  7. Antonio Torralba
  8. Chen Change Loy

In Computer Graphics and Computer-Aided Design

  1. Eli Shechtman
  2. Richard Zhang
  3. Alexei A. Efros
  4. Antonio Torralba
  5. Ziwei Liu
  6. Tali Dekel
  7. William T. Freeman
  8. Aleksander Holynski

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Ming-Hsuan Yang
  3. Dacheng Tao
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Chen Change Loy
  8. Andrea Vedaldi

In Generative Models

  1. Eli Shechtman
  2. Karsten Kreis
  3. Arash Vahdat
  4. Richard Zhang
  5. Jun-Yan Zhu
  6. Ming-Hsuan Yang
  7. Alexei A. Efros
  8. Luc Van Gool

Citations in the corpus