Zero-shot Image-to-Image Translation

By Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, Jun-Yan Zhu. International Conference on Computer Graphics and Interactive Techniques, 2023

653 citations recorded by the source corpus.

Large-scale text-to-image generative models have shown their remarkable ability to synthesize diverse, high-quality images. However, directly applying these models for real image editing remains challenging for two reasons. First, it is hard for users to craft a perfect text prompt depicting every visual detail in the input image. Second, while existing models can introduce desirable changes in certain regions, they often dramatically alter the input content and introduce unexpected changes in unwanted regions. In this work, we introduce pix2pix-zero, an image-to-image translation method that can preserve the original image’s content without manual prompting. We first automatically discover editing directions that reflect desired edits in the text embedding space. To preserve the content structure, we propose cross-attention guidance, which aims to retain the cross-attention maps of the input image throughout the diffusion process. Finally, to enable interactive editing, we distill the diffusion model into a fast conditional GAN. We conduct extensive experiments and show that our method outperforms existing and concurrent works for both real and synthetic image editing. In addition, our method does not need additional training for these edits and can directly use the existing pre-trained text-to-image diffusion model.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/daf61010eee0fbf6f9bab7db71c395ffca6f3ff3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “…Rombach et al., 2022) has taken center stage, ﬁnding applications in various image processing domainsuchasimageoutpainting(Rameshetal.,2022),editing (Zhang & Agrawala, 2023; Mokady et al., 2022; Parmar et al., 2023; Huang et al., 2023) and enhancement (Saharia et al., 2022; Wang et al., 2023).”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, background): “Besides, diffusion models also show great success in text-to-video generation [18, 47, 52], 3D Avatar generation [39, 53], image to image translation [37], image manipulation [4, 26], music generation [23], and even drug discovery [55].”
An Edit Friendly DDPM Noise Space: Inversion and Manipulations (2023, methodology): “Particularly, the CLIP similarity and LPIPS distance achieved by CycleDiffusion on the modified ImageNet-R-TI2I dataset are plotted in Fig.”
Step1X-Edit: A Practical Framework for General Image Editing (2025, neutral): “With the introduction of DiT architectures [41], diffusion models have made significant strides in generalization, image quality, and knowledge capacity, becoming the predominant architecture in modern image generation [1, 5, 7].”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, background): “We have noticed some recent efforts [30, 2] in this area.”
DragDiffusion: Harnessing Diffusion Models for Interactive Point-Based Image Editing (2023, positive): “In contrast to previous methods [3, 10, 34, 53], which apply gradient descent on latents of multiple diffusion steps, our approach focuses on optimizing the latent of one appropriately selected step to conveniently achieve the desired editing results.”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “Based on these models, advanced image and video editing have been achieved through finetuning the model [41, 67, 5, 23, 61, 29] or controlling the inference process [30, 13, 34, 10, 35, 7, 31, 3].”
Unleashing Text-to-Image Diffusion Models for Visual Perception (2023, background): “a cat is sitting on a table verse and high-fidelity images with high customizabil-ity [45, 19, 38, 6], attracting growing attention from both the research community and the public eye.”
FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, methodology): “We conduct the quantitative evaluation using the trained CLIP [34] model as previous methods [9,38,52].”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, methodology): “Text-guided control methods focus on adjusting prompts, manipulating CLIP features, and modifying cross-attention [7, 10, 20, 27, 40, 41, 57, 63, 66].”
FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, neutral): “…n } , a list of reference subject images S = { s 1 , s 2 , . . . s m } , and an index list indicating which subject corresponds to which word in the text prompt I = { i 1 , i 2 , . . . i m } , i j ∈ 1 , 2 , . . . , n , we ﬁrst encode the text prompt P and reference subjects S into embeddings…”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “Fine-tuning aims to adapt the weights of a pretrained generative model to new domains [73, 46, 72, 41, 79, 34, 47, 80, 30, 35, 24, 44], downstream tasks [71, 54, 78], and test images [6, 53, 48, 31, 25, 49].”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, methodology): “For text-conditioned image editing, existing meth-ods [10, 35, 21] achieve impressive editing effects in image translation, stylization, and appearance replacement while keeping the input structure and scene layout unchanged.”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “Regarding image editing, pix2pix-zero [38] and prompt-to-prompt editing [19] follow instructional texts by manipulating cross-attention maps.”
Implicit Diffusion Models for Continuous Super-Resolution (2023, neutral): “In image super-resolution, regression-based methods, such as EDSR [23], RRDB [45], and SWinIR [21], directly learn a mapping from LR to HR images with an MSE loss.”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Diffusion-based image editing Diffusion models have also shown great potential for semantic editing [33, 3, 2, 31, 79, 37, 72, 68, 63, 42, 43, 5, 71].”
Emergent Correspondence from Image Diffusion (2023, background): “For instance, they could convert a dog to a cat without changing its pose or context [62].”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “Several promising research directions include: • Exploring applications of in-context generation capabilities of D REAM LLM to complex tasks such as image-to-image translation (Isola et al., 2017; Zhang et al., 2023c;d; Parmar et al., 2023).”
DIRE for Diffusion-Generated Image Detection (2023, background): “Since then, more and more researchers turn their attention to diffusion models for improving the architectures [10, 37], accelerating sampling speed [32, 42, 23, 24], exploring downstream tasks [16, 31, 1, 33], and etc .”
Latte: Latent Diffusion Transformer for Video Generation (2024, neutral): “In the assessment of quantitative comparisons, we employ three evaluation metrics: Fr´echet Video Distance (FVD) ) (Parmar et al, 2021), and Inception Score (IS) (Saito et al, 2017).”