Prompt-to-Prompt Image Editing with Cross Attention Control
By Amir Hertz, Ron Mokady, Josh Tenenbaum, Kfir Aberman, Yael Pritch, Daniel Cohen-Or. 2022
2,848 citations recorded by the source corpus.
Recent large-scale text-driven synthesis models have attracted much attention thanks to their remarkable capabilities of generating highly diverse images that follow given text prompts. Such text-based synthesis methods are particularly appealing to humans who are used to verbally describe their intent. Therefore, it is only natural to extend the text-driven image synthesis to text-driven image editing. Editing is challenging for these generative models, since an innate property of an editing technique is to preserve most of the original image, while in the text-based models, even a small modification of the text prompt often leads to a completely different outcome. State-of-the-art methods mitigate this by requiring the users to provide a spatial mask to localize the edit, hence, ignoring the original structure and content within the masked region. In this paper, we pursue an intuitive prompt-to-prompt editing framework, where the edits are controlled by text only. To this end, we analyze a text-conditioned model in depth and observe that the cross-attention layers are the key to controlling the relation between the spatial layout of the image to each word in the prompt. With this observation, we present several applications which monitor the image synthesis by editing the textual prompt only. This includes localized editing by replacing a word, global editing by adding a specification, and even delicately controlling the extent to which a word is reflected in the image. We present our results over diverse images and prompts, demonstrating high-quality synthesis and fidelity to the edited prompts.
Subfields: Mechanical Engineering, Generative Models, Multimodal Learning, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Mechanical Engineering
In Generative Models
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Alexei A. Efros
- Daniel Cohen-Or
- Arash Vahdat
- Ming-Hsuan Yang
- Karsten Kreis
In Multimodal Learning
In Human-Computer Interaction
Citations in the corpus
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “…new edits and make correct associations between visual changes and text instructions is limited by the human-written instructions used to fine-tune GPT-3 [7], by the ability of GPT-3 to create instructions and modify captions, and by the ability of Prompt-to-Prompt [17] to modify generated images.”
- An Edit Friendly DDPM Noise Space: Inversion and Manipulations (2023, methodology): “[7, 17, 25, 4, 27, 19]) rely on an approximate inversion method for the denoising diffusion implicit model (DDIM) scheme [24], which is a deterministic sampling process that maps a single initial noise vector into a generated image.”
- KNN-Diffusion: Image Generation via Large-Scale Retrieval (2022, background): “Recently, several works (Bar-Tal et al., 2022; Hertz et al., 2022; Gal et al., 2022) have made progress with local manipulations without relying on user edited masks.”
- eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, methodology): “Prompt-to-prompt tuning can achieve image editing by modifying the textual prompt used to produce the same image without having the user provide object-specific segmentation masks [22].”
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, methodology): “Na¨ıvely leveraging (cid:15) θ for video editing, by applying an image editing method on each frame independently (e.g., [12, 51, 29, 56]), results in content inconsistencies across frames (e.g., Fig.”
- Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, methodology): “In the context of text-based image generation, the cross-attention maps have been considered a natural explanation for the model [Hertz et al. 2022].”
- Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “These models enable various image manipulation tasks [3,22,30,52] previously unassailable, allowing image editing for general images with the guidance of text prompt.”
- Muse: Text-To-Image Generation via Masked Generative Transformers (2023, methodology): “A number of techniques have been developed on diffusion models to enable editing, personalization and inversion to token space (Gal et al., 2022a; Meng et al., 2021; Ruiz et al., 2022; Kawar et al., 2022; Brooks et al., 2022; Hertz et al., 2022; Mokady et al., 2022).”
- Step1X-Edit: A Practical Framework for General Image Editing (2025, neutral): “Based on the GEdit-Bench , we evaluated a diverse range of image editing algorithms, covering state-of-the-art open-source solutions such as Instruct-Pix2Pix [8], MagicBrush [66], AnyEdit [64], OmniGen [61], as well as proprietary algorithms like GPT-4.”
- BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing (2023, methodology): “The model can also serve as a foundation subject-driven textto-image generation model and supports applications such as controlled generation and image editing, when combined with techniques such as ControlNet [1] and prompt-to-prompt [2].”
- Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “4 Personalization,Customization, and Control of Pretrained Diffusion Model Because state-of-the-art image diffusion models are dominated by text-to-image methods, the most straight-forward ways to enhance the control over a diffusion model are often text-guided [38, 24, 2, 3, 23, 43, 16].”
- InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, background): “By properly exploiting their latent structure, prior works succeeded in image editing [33, 79, 80, 81, 82, 83, 84], semantic control [55, 56, 85], disentangled control direction discovery [86, 87, 88, 89], etc..”
- Unleashing Text-to-Image Diffusion Models for Visual Perception (2023, background): “Recent work by [19] has witnessed a clear visual-text correlation in the large text-to-image diffusion models, which motivates us to study whether the pre-trained knowledge can be exploited to facilitate downstream visual perception tasks.”
- MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, background): “Recent works (Tumanyan et al., 2022; Hertz et al., 2022) rely on architectural properties and insights about the internal features of the pretrained model, and tailor image editing techniques accordingly.”
- JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models (2023, methodology): “Furthermore, we present generation examples as evidence of JEN-1’s proficiency in capturing music-related semantics and exhibiting exceptional controllability incorporated with editing techniques like prompt2prompt (Hertz et al., 2022).”
- Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators (2023, methodology): “With the rise of text-guided image editing methods such as Prompt2Prompt [ 9], Instruct-Pix2Pix [2], SDEdit [20], etc., text-guided video editing approaches emerged [1, 16, 42].”
- MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, methodology): “Unlike previous works [10, 21, 3] that usually operate on cross-attention in T2I models, we propose to convert existing self-attention to mutual self-attention, so that it can query correlated local structures and textures from a source image for consistency.”
- AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, neutral): “[47] uses paired image data generated by [14] to train a text-based image editing model.”