DiffEdit: Diffusion-based semantic image editing with mask guidance

By Guillaume Couairon, Jakob Verbeek, Holger Schwenk, Matthieu Cord. International Conference on Learning Representations, 2022

779 citations recorded by the source corpus.

Image generation has recently seen tremendous advances, with diffusion models allowing to synthesize convincing images for a large variety of text prompts. In this article, we propose DiffEdit, a method to take advantage of text-conditioned diffusion models for the task of semantic image editing, where the goal is to edit an image based on a text query. Semantic image editing is an extension of image generation, with the additional constraint that the generated image should be as similar as possible to a given input image. Current editing methods based on diffusion models usually require to provide a mask, making the task much easier by treating it as a conditional inpainting task. In contrast, our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited, by contrasting predictions of a diffusion model conditioned on different text prompts. Moreover, we rely on latent inference to preserve content in those regions of interest and show excellent synergies with mask-based diffusion. DiffEdit achieves state-of-the-art editing performance on ImageNet. In addition, we evaluate semantic image editing in more challenging settings, using images from the COCO dataset as well as text-based generated images.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/064ccebc03d3afabaae30fe29a457c1cfcdff7e3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, background): “Avrahami et al., 2022c; Choi et al., 2021; Mokady et al., 2022; Couairon et al., 2022; Kong et al., 2023; Kwon et al., 2022) or by performing lightweight model finetuning (Ruiz et al., 2022; Kawar et al., 2022; Kim et al., 2022; Valevski et al., 2022).”
An Edit Friendly DDPM Noise Space: Inversion and Manipulations (2023, methodology): “[7, 17, 25, 4, 27, 19]) rely on an approximate inversion method for the denoising diffusion implicit model (DDIM) scheme [24], which is a deterministic sampling process that maps a single initial noise vector into a generated image.”
SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model (2022, neutral): “, unconditional image generation [7, 8, 25, 26], text-toimage generation [18–21], video generation [6], image inpainting [1,2,14,16], image translation [15,27,30], and image editing [4, 5, 10].”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “DiffEdit [11] introduces automatically generated masks to assist text-driven image editing.”
Composer: Creative and Controllable Image Synthesis with Composable Conditions (2023, methodology): “, 2023), segmentation maps (Rombach et al., 2021; Wang et al., 2022b; Couairon et al., 2022), a composition of multiple text descriptions (Liu et al.”
Style Injection in Diffusion: A Training-Free Approach for Adapting Large-Scale Diffusion Models for Style Transfer (2023, methodology): “Recent advances in Diffusion Models (DMs) have led to breakthroughs in various generative applications such as text-to-image synthesis [30, 33, 35] and image or video editing [2, 4, 6, 14, 19, 41, 48].”
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, neutral): “Recent diffusion-based image editing models [27, 15, 5, 50, 22, 45] can process each individual frame in a video, but this produces inconsistency between frames due to the lack of temporal awareness in the model.”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, methodology): “With inferred masks, DiffEdit [135] replaces the region of interest with pixels corresponding to the query text.”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “This can be circumvented by providing or generating masks for the objects of interest [7,31].”
The Stable Signature: Rooting Watermarks in Latent Diffusion Models (2023, positive): “Hence it is to our knowledge compatible with all LDM-based generative methods [7, 13, 62, 71, 104].”
FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, methodology): “Firstly, in contrast to the straightforward DDIM inversion technique, our approach captures intermediate attention maps during inversion, which effectively retain both structural and motion information.”
EDICT: Exact Diffusion Inversion via Coupled Transformations (2022, neutral): “DiffEdit [3] edits real/synthetic images using automatically generated masks for regions of an input image that should be edited given a text query.”
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting (2022, methodology): “There has been much recent work on text-guided image inpainting [1, 3, 5, 7, 19, 27, 31, 33].”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, methodology): “In addition, several other methods [11, 16, 25, 62] focus on accessing and editing features for downstream tasks such as stylization and image editing.”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Similar image-editing capabilities can also be achieved by fine-tuning the model parameters [37, 78] or automatically finding editing masks with the denoiser [12].”