StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery

By Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, Dani Lischinski. 2021

1,474 citations recorded by the source corpus.

Inspired by the ability of StyleGAN to generate highly realistic images in a variety of domains, much recent work has focused on understanding how to use the latent spaces of StyleGAN to manipulate generated and real images. However, discovering semantically meaningful latent manipulations typically involves painstaking human examination of the many degrees of freedom, or an annotated collection of images for each desired manipulation. In this work, we explore leveraging the power of recently introduced Contrastive Language-Image Pre-training (CLIP) models in order to develop a text-based interface for StyleGAN image manipulation that does not require such manual effort. We first introduce an optimization scheme that utilizes a CLIP-based loss to modify an input latent vector in response to a user-provided text prompt. Next, we describe a latent mapper that infers a text-guided latent manipulation step for a given input image, allowing faster and more stable text-based manipulation. Finally, we present a method for mapping text prompts to input-agnostic directions in StyleGAN’s style space, enabling interactive text-driven image manipulation. Extensive results and comparisons demonstrate the effectiveness of our approaches.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “Since CLIP provides a score of how close an image is to a caption, several works have used it to steer generative models like GANs towards a user-defined text caption (Galatolo et al., 2021; Patashnik et al., 2021; Murdock, 2021; Gal et al., 2021).”
Generative Adversarial Networks (2021, neutral): “Style transfer using GANs has shown promising results in various domains, s.a. fashion design, art, and photography [19].”
Towards Language-Free Training for Text-to-Image Generation (2021, background): “The idea of multimodal feature space also inspires some recent works on generative models [9, 10, 33, 35].”
Projected GANs Converge Faster (2021, neutral): “Pretrained models can be used as a guiding mechanism to disentangle causal generative factors [54], for text-driven image manipulation [44], matching the generator activations to inverted classifiers [19, 56], or to generate images via gradient ascent in the latent space of a generator [41].”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, neutral): “The disciminative model can operate on either the image space [66, 70, 27] or the latent space [80, 98].”
Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “This could be used to condition pre-trained models in a plethora of ways, for example by conditioning an image generator with a text caption using a noisy version of CLIP [42], similar to recent methods that guide GANs using text prompts [15, 40, 2].”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, methodology): “The image search domain used by these methods could be the latent codes of a pre-trained GAN model (e.g. BigGAN [1], StyleGAN [14, 24], SWAGAN [4]), the codebook of a VQGAN model [7], Diffusion Denoising Models [3], structured representations such as a set of strokes (ClipDraw [13]) or triangles [6]”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “There exists specialized techniques for this task [3,37,44,57], but here we aim to showcase the quality and applicability of our latent space by using the simplest linear operation.”
StyleGAN-NADA (2021, methodology): “By traversing this intermediate latent space, W , or by mixing different w codes across different network layers, prior work demonstrated fine-grained control over semantic properties in generated images [2, 13, 30, 36].”
GAN Inversion: A Survey (2021, methodology): “image understanding [134], [135], multimodal learning [93],”
ClipCap: CLIP Prefix for Image Captioning (2021, background): “Many works have already used CLIP successfully for computer vision tasks that require the understanding of some auxiliary text, such as generating or editing an image based on a natural language condition [5, 14, 28].”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, background): “and as it does not employ an intermediate latent space which is essential for GAN-based image editing [Abdal et al. 2021; Collins et al. 2020; Patashnik et al. 2021; Wu et al. 2021].”
Alias-Free Generative Adversarial Networks (2021, positive): “The changes in expression were realized by applying the “global directions” method of StyleCLIP [46], using the prompts “angry face”, “laughing face”, “kissing face”, “sad face”, “singing face”, and “surprised face”.”
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance (2022, background): “There are other examples in the literature of generative models that can perform style transfer [36], image inpainting [38, 33], and other types of image manipulation [34], but we note that each of these represent distinct tasks from open domain semantic image editing.”
Multi-Concept Customization of Text-to-Image Diffusion (2022, background): “Several works aim at leveraging the capabilities of generative models, such as GANs [2–4, 49, 85] or diffusion models [10,22,30,45] towards editing.”
Zero-Shot Text-Guided Object Generation with Dream Fields (2021, methodology): “Recent work from Mario Klingemann (@quasimondo) and [43] have shown how CLIP can be used to guide GAN models like StyleGAN [27].”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, methodology): “Recent works have attempted to overcome these issues by first training an unconditional generator, and then converting it to a conditional model with a small cost [26, 20, 1, 42].”