Blended Latent Diffusion

By Omri Avrahami, Ohad Fried, Dani Lischinski. ACM Transactions on Graphics, 2022

568 citations recorded by the source corpus.

The tremendous progress in neural image generation, coupled with the emergence of seemingly omnipotent vision-language models has finally enabled text-based interfaces for creating and editing images. Handling generic images requires a diverse underlying generative model, hence the latest works utilize diffusion models, which were shown to surpass GANs in terms of diversity. One major drawback of diffusion models, however, is their relatively slow inference time. In this paper, we present an accelerated solution to the task of local text-driven editing of generic images, where the desired edits are confined to a user-provided mask. Our solution leverages a text-to-image Latent Diffusion Model (LDM), which speeds up diffusion by operating in a lower-dimensional latent space and eliminating the need for resource-intensive CLIP gradient calculations at each diffusion step. We first enable LDM to perform local image edits by blending the latents at each step, similarly to Blended Diffusion. Next we propose an optimization-based solution for the inherent inability of LDM to accurately reconstruct images. Finally, we address the scenario of performing local edits using thin masks. We evaluate our method against the available baselines both qualitatively and quantitatively and demonstrate that in addition to being faster, it produces more precise results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ef669bb2d0a3e957a91c1dde85ce01c6984ad7d6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, background): “To overcome subject modification, several works [3, 42] assume a user-provided mask to restrict the modified area.”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Diffusion-based image editing Diffusion models have also shown great potential for semantic editing [33, 3, 2, 31, 79, 37, 72, 68, 63, 42, 43, 5, 71].”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “For example, blended diffusion [2, 3] performs inpainting with natural language descriptions.”
Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation (2023, methodology): “Based on it, a multitude of image editing methods have emerged, including model fine-tuning for customized object generation [33], imageto-image translation [23], image inpainting [1], and object editing [12].”
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures (2022, background): “Previous works [1,45] showed that super-pixels in Z depend mainly on individual patches in the output image.”
TEXTure: Text-Guided Texturing of 3D Shapes (2023, methodology): “Although current text-to-image models generate highquality results when conditioned on a text prompt or depth map, editing an existing image or injecting objects specified by a few exemplar images remains challenging [2,47].”
Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos (2023, background): “Such AI generative systems excel in various aspects, including rendering realistic and diverse appearances [20, 41, 16], powerful editing [9, 23], composition [1], generalization capabilities, and creating new art that satisﬁes people’s imagination [20, 41].”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, background): “…object detection and segmentation [41, 37, 86], object tracking [67, 84], image generation [75, 54, 48, 45, 23, 14, 52, 22, 82, 44], image editing [42, 1, 2, 53, 21], human-centric perception and understanding [72, 71, 73, 70, 69, 4, 33, 74], and human-centric motion generation [39, 6, 32, 61, 5].”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, background): “For example, the works [19, 1] require extra masks to edit local regions of the image; [13] can edit global aspects of the image by changing the text prompt directly, but cannot modify local details; [10, 35] use cross-attention or spatial features to edit both global and local aspects of the image…”
Diffusion Models in Vision: A Survey (2022, methodology): “[114] apply latent diffusion models to edit images locally, using text.”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, methodology): “…methods are generally limited to global edits [Chefer et al. 2022a; Crowson et al. 2022; Gal et al. 2022b; Kwon and Ye 2022], several works introduce a user-provided mask to specify the region that should be altered [Avrahami et al. 2022a; Bau et al. 2021; Couairon et al. 2022; Nichol et al. 2021].”
SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model (2022, unknown): “, unconditional image generation [7, 8, 25, 26], text-toimage generation [18–21], video generation [6], image inpainting [1,2,14,16], image translation [15,27,30], and image editing [4, 5, 10].”
FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, methodology): “4, we obtain a binary mask M t by thresholding the cross-attention map of the edited words during inversion by a constant τ [2,3].”
Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation (2022, background): “, a GAN [7, 15, 27, 32] or a diffusion model [2, 3, 22, 25].”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, background): “The above works [132], [133], [134] require a manually designed mask so that the model can tell which part to edit.”
KNN-Diffusion: Image Generation via Large-Scale Retrieval (2022, background): “The majority of previous works in the task of image manipulation either rely on user-provided masks (Nichol et al., 2021; Avrahami et al., 2022b;a), or are limited to global editing (Crowson et al., 2022; Kim et al., 2022).”
Video-P2P: Video Editing with Cross-Attention Control (2023, unknown): “Generative models have demonstrated impressive performance in image editing, with approaches ranging from GANs [12, 27, 28, 44] to diffusion models [1, 19].”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, methodology): “To overcome this, several works [28, 4] assume that the user provides a mask to restrict the area in which the changes are applied.”