Blended Diffusion for Text-driven Editing of Natural Images

By Omri Avrahami, Dani Lischinski, Ohad Fried. 2021

1,278 citations recorded by the source corpus.

Natural language offers a highly intuitive interface for image editing. In this paper, we introduce the first solution for performing local (region-based) edits in generic natural images, based on a natural language description along with an ROI mask. We achieve our goal by leveraging and combining a pretrained language-image model (CLIP), to steer the edit towards a user-provided text prompt, with a denoising diffusion probabilistic model (DDPM) to generate natural-looking results. To seamlessly fuse the edited region with the unchanged parts of the image, we spatially blend noised versions of the input image with the local text-guided diffusion latent at a progression of noise levels. In addition, we show that adding augmentations to the diffusion process mitigates adversarial results. We compare against several baselines and related methods, both qualitatively and quantitatively, and show that our method outperforms these solutions in terms of overall realism, ability to preserve the background and matching the text. Finally, we show several text-driven editing applications, including adding a new object to an image, removing/replacing/altering existing objects, background replacement, and image extrapolation.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “For our results, we follow Avrahami et al. (2021) and use CLIP to select the best of 64 samples.”
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023, methodology): “These methods can be roughly divided into two categories: (1) zero-shot editing [2, 1, 24], these pipeline methods require massive amount of manual tuning of its hyperparameters to produce reasonable results.”
Style Injection in Diffusion: A Training-Free Approach for Adapting Large-Scale Diffusion Models for Style Transfer (2023, positive): “Recent advances in Diffusion Models (DMs) have led to breakthroughs in various generative applications such as text-to-image synthesis [30, 33, 35] and image or video editing [2, 4, 6, 14, 19, 41, 48].”
DeepCache: Accelerating Diffusion Models for Free (2023, background): “A large number of attractive applications have been facilitated with diffusion models, including but not limited to image editing [2, 22, 40], image super-enhancing [28, 53], image-to-image translation [7, 51], text-to-image generation [43, 47, 48, 52] and text-to-3D generation [32, 37, 45].”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, methodology): “To handle this situation, the work in [1; 16] turn to guide the intermediate variable x̃0 to control the generation process of diffusion models.”
Blended Latent Diffusion (2022, methodology): “As in previous works [Razavi et al. 2019; Ramesh et al. 2021; Avrahami et al. 2021], we found it beneficial to generate multiple predictions, rank them, and retrieve the best ones.”
DiffusionDet: Diffusion Model for Object Detection (2022, background): “Diffusion models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications…”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, methodology): “Other works [5, 22] exploit the recent Diffusion models [17, 39, 41, 17, 40, 36], which achieve state-of-the-art generation quality over highly diverse datasets, often surpassing GANs [10].”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, background): “Text-based Generation has seen significant advances due to large-scale image-text datasets [73, 72, 14, 71] and scalable generative model architectures [16, 66, 60, 32], enabling synthesis of novel images from text [20, 3, 54].”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, background): “…object detection and segmentation [41, 37, 86], object tracking [67, 84], image generation [75, 54, 48, 45, 23, 14, 52, 22, 82, 44], image editing [42, 1, 2, 53, 21], human-centric perception and understanding [72, 71, 73, 70, 69, 4, 33, 74], and human-centric motion generation [39, 6, 32, 61, 5].”
ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting (2023, background): “Furthermore, it has also demonstrated great potential in solving several downstream low-level vision tasks, including image editing [4, 5], image inpainting [6, 7], image colorization [8, 9].”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, background): “The above works [132], [133], [134] require a manually designed mask so that the model can tell which part to edit.”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, positive): “In this study, we extend our exploration beyond the realm of content creation and examine the potential beneﬁts of using the diffusion prior for super-resolution (SR).”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Compared with text-conditioned inpainting methods [3, 6, 51] that apply a single concept to an image region, “paint-with-words” can generate the whole image containing multiple concepts in a single pass from scratch, without the need to start from an input image.”
Emu Edit: Precise Image Editing via Recognition and Generation Tasks (2023, neutral): “Blended Diffusion [1] modifies the diffusion step by blending the input image in the unmasked regions.”
DIRE for Diffusion-Generated Image Detection (2023, background): “Since then, more and more researchers turn their attention to diffusion models for improving the architectures [10, 37], accelerating sampling speed [32, 42, 23, 24], exploring downstream tasks [16, 31, 1, 33], and etc .”