Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation

By Narek Tumanyan, Michal Geyer, Shai Bagon, Tali Dekel. Computer Vision and Pattern Recognition, 2022

1,092 citations recorded by the source corpus.

Large-scale text-to-image generative models have been a revolutionary breakthrough in the evolution of generative AI, synthesizing diverse images with highly complex visual concepts. However, a pivotal challenge in leveraging such models for real-world content creation is providing users with control over the generated content. In this paper, we present a new framework that takes text-to- image synthesis to the realm of image-to-image translation - given a guidance image and a target text prompt as input, our method harnesses the power of a pre-trained text-to-image diffusion model to generate a new image that complies with the target text, while preserving the semantic layout of the guidance image. Specifically, we observe and empirically demonstrate that fine-grained control over the generated structure can be achieved by manipulating spatial features and their self-attention inside the model. This results in a simple and effective approach, where features extracted from the guidance image are directly injected into the generation process of the translated image, requiring no training or fine-tuning. We demonstrate high-quality results on versatile text-guided image translation tasks, including translating sketches, rough drawings and animations into realistic images, changing the class and appearance of objects in a given image, and modifying global qualities such as lighting and color.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Literature and Literary Theory

View Paper: https://www.semanticscholar.org/paper/b000d6865db824af1563708fb7a545ddd65c6b3a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Literature and Literary Theory

Citations in the corpus

Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “Prompt-based image editing [10, 33, 85] provides practical tools to manipulate images with prompts.”
Pix2Video: Video Editing using Image Diffusion (2023, neutral): “In another line of work, methods have focused on editing images while preserving structures via attention layer manipulation [15, 52], spatial guidance [7], or per-instance finetuning [24].”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, methodology): “In addition, several other methods [11, 16, 25, 62] focus on accessing and editing features for downstream tasks such as stylization and image editing.”
FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, methodology): “Firstly, in contrast to the straightforward DDIM inversion technique, our approach captures intermediate attention maps during inversion, which effectively retain both structural and motion information.”
FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, positive): “Given a text prompt P = { w 1 , w 2 , . . . w n } , a list of reference subject images S = { s 1 , s 2 , . . . s m } , and an index list indicating which subject corresponds to which word in the text prompt I = { i 1 , i 2 , . . . i m } , i j ∈ 1 , 2 , . . . , n , we ﬁrst encode the text prompt P…”
DragDiffusion: Harnessing Diffusion Models for Interactive Point-Based Image Editing (2023, background): “, 2022), many diffusion-based image editing methods have been proposed (Hertz et al., 2022; Cao et al., 2023; Mao et al., 2023; Kawar et al., 2023; Parmar et al., 2023; Liew et al., 2022; Mou et al., 2023; Tumanyan et al., 2023; Brooks et al., 2023; Meng et al., 2021; Bar-Tal et al., 2022).”
Emergent Correspondence from Image Diffusion (2023, background): “While diffusion models are primarily models for image synthesis, a key observation is that these models produce good results for image-to-image translation [54, 83] and image editing [8].”
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, neutral): “2) Plug-and-Play [45]: a cutting-edge image editing model that can edit each frame of a video individually.”
Zero-shot Image-to-Image Translation (2023, background): “2022], and Plug&Play [Tumanyan et al. 2023] also attempt the task of text-based image translation.”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, methodology): “…source image with source prompt, synthesis results with the proposed MasaCtrl method, synthesis results from target prompt with the same random noise of source image, synthesis/editing results with existing methods P2P [10], SDEdit [16], PnP [35], and Imagic [12]. with Adapter and Anything”
MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, methodology): “(ii) On the other side of the spectrum, numerous methods proposed to implicitly control the generated content by manipulating the generation process of a pre-trained model (Kwon & Ye, 2022; Meng et al., 2021; Tumanyan et al., 2022; Hertz et al., 2022; Figure 2.”
Diffusion Self-Guidance for Controllable Image Generation (2023, methodology): “…the self and cross-attention maps (cid:8) A i,t ∈ R H i × W i × K (cid:9) often encode structural information [11] about object position and shape, while the network activations (cid:8) Ψ i,t ∈ R H i × W i × D i allow for maintaining coarse appearance [36] when extracted from appropriate layers.”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, background): “In image-to-image translation task, [127] also works on the semantic features in the diffusion latent space, and finds that manipulating spatial features and their self-attention inside the model can control the image translation process.”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Diffusion-based image editing Diffusion models have also shown great potential for semantic editing [33, 3, 2, 31, 79, 37, 72, 68, 63, 42, 43, 5, 71].”
Emu Edit: Precise Image Editing via Recognition and Generation Tasks (2023, methodology): “Finally, to support evaluation for methods that require input and output cap-tions [8, 21], we additionally collect an input caption and output caption.”
TEXTure: Text-Guided Texturing of 3D Shapes (2023, methodology): “Finally, in the spirit of diffusion-based image editing [20–22, 48 ], we show We evaluate TEXTure and show its effectiveness for texture generation, transfer, and editing.”
Video-P2P: Video Editing with Cross-Attention Control (2023, positive): “Diffusion models have demonstrated outstanding generation capabilities in this area [3, 13, 19, 40].”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, methodology): “Middle: Frames and feature visualization for an edited video obtained by applying an image editing method ([51]) on each frame; inconsistent patterns in RGB are also evident in the feature space (e.g., on the dog’s body).”