DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation

By Gwanghyun Kim, Taesung Kwon, Jong Chul Ye. 2021

900 citations recorded by the source corpus.

Recently, GAN inversion methods combined with Contrastive Language-Image Pretraining (CLIP) enables zeroshot image manipulation guided by text prompts. However, their applications to diverse real images are still difficult due to the limited GAN inversion capability. Specifically, these approaches often have difficulties in reconstructing images with novel poses, views, and highly variable contents compared to the training data, altering object identity, or producing unwanted image artifacts. To mitigate these problems and enable faithful manipulation of real images, we propose a novel method, dubbed DiffusionCLIP, that performs textdriven image manipulation using diffusion models. Based on full inversion capability and high-quality image generation power of recent diffusion models, our method performs zeroshot image manipulation successfully even between unseen domains and takes another step towards general application by manipulating images from a widely varying ImageNet dataset. Furthermore, we propose a novel noise combination method that allows straightforward multi-attribute manipulation. Extensive experiments and human evaluation confirmed robust and superior manipulation performance of our methods compared to the existing baselines. Code is available at https://github.com/gwang-kim/DiffusionCLIP.git

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-Concept Customization of Text-to-Image Diffusion (2022, background): “Several works aim at leveraging the capabilities of generative models, such as GANs [2–4, 49, 85] or diffusion models [10,22,30,45] towards editing.”
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, methodology): “More following work are proposed using transformers [10, 11] or diffusion models [18, 3, 5, 17, 19, 23, 25] to generate high-fidelity images from text.”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “Similarly, GANs and diffusion models employ CLIP and its variants for text-driven image manipulation and editing [220], [221].”
Muse: Text-To-Image Generation via Masked Generative Transformers (2023, methodology): “This method works directly on the (tokenized) image and does not require “inverting” the generative process, in contrast with recent zero-shot image editing techniques leveraging generative models (Gal et al., 2022b; Patashnik et al., 2021; Kim et al., 2022; Mokady et al., 2022).”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, background): “For example, the works [19, 1] require extra masks to edit local regions of the image; [13] can edit global aspects of the image by changing the text prompt directly, but cannot modify local details; [10, 34] use cross-attention or spatial features to edit both global and local aspects of the image ”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, background): “…diffusion models have demonstrated superior performance across a range of generation tasks in various modalities, including text-to-image generation [40, 43, 41, 17], image manipulation [26, 34, 4, 56], video synthesis [18, 47, 22], text generation [28, 16, 59], 3D avatar synthesis [39, 53], etc.”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, background): “Other works [4, 31] exploit the recent diffusion models [25, 25, 45, 58, 60, 62–66], which achieve state-of-the-art generation quality over highly diverse datasets, often surpassing GANs [15].”
PromptTTS 2: Describing and Generating Voices with Text Prompt (2023, methodology): “, 2020), which has demonstrated a robust capability in modeling multimodal distributions and complex data spaces (Kim et al., 2022; Ramesh et al., 2022; Ho et al., 2022; Nichol & Dhariwal, 2021; Leng et al., 2022).”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “4 Personalization,Customization, and Control of Pretrained Diffusion Model Because state-of-the-art image diffusion models are dominated by text-to-image methods, the most straight-forward ways to enhance the control over a diffusion model are often text-guided [38, 24, 2, 3, 23, 43, 16].”
KNN-Diffusion: Image Generation via Large-Scale Retrieval (2022, background): “The majority of previous works in the task of image manipulation either rely on user-provided masks (Nichol et al., 2021; Avrahami et al., 2022b;a), or are limited to global editing (Crowson et al., 2022; Kim et al., 2022).”
Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, positive): “Experiment Setup: Given an input image, we ﬁrst perform DDIM inversion [70, 41] (with 50 timesteps) us-ing Stable Diffusion 2.1 and different captions as prompts: BLIP [47] generated caption, human-reﬁned BLIP generated caption, “a photo of { correct-class-name } , a type of pet” and “a photo of {…”
MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, background): “, 2022) or by performing lightweight model finetuning (Ruiz et al., 2022; Kawar et al., 2022; Kim et al., 2022; Valevski et al., 2022).”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, background): “Conditional diffusion models (CDMs) with diverse conditions have been proposed, such as text [20, 1, 13, 23, 33, 37, 32, 35, 22, 29], class labels [10], degraded images [5, 6, 7, 8, 19, 24, 39, 41, 44, 34, 36, 45], segmentation maps [28, 49], landmarks [28, 49], hand-drawn sketches [28, 49], style…”
An Edit Friendly DDPM Noise Space: Inversion and Manipulations (2023, methodology): “DiffusionCLIP [12] uses model fine-tuning based on a CLIP loss with a target text.”
Imagic: Text-Based Real Image Editing with Diffusion Models (2022, methodology): “Many methods were developed for text-based image editing, showing promising results and continually improving [8, 10, 33].”