StyleGAN-NADA

By Rinon Gal, Or Patashnik, Haggai Maron, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or. 2021

319 citations recorded by the source corpus.

Can a generative model be trained to produce images from a specific domain, guided only by a text prompt, without seeing any image? In other words: can an image generator be trained "blindly"? Leveraging the semantic power of large scale Contrastive-Language-Image-Pre-training (CLIP) models, we present a text-driven method that allows shifting a generative model to new domains, without having to collect even a single image. We show that through natural language prompts and a few minutes of training, our method can adapt a generator across a multitude of domains characterized by diverse styles and shapes. Notably, many of these modifications would be difficult or infeasible to reach with existing methods. We conduct an extensive set of experiments across a wide range of domains. These demonstrate the effectiveness of our approach, and show that our models preserve the latent-space structure that makes generative models appealing for downstream tasks. Code and videos available at: stylegan-nada.github.io/

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, methodology): “The image search domain used by these methods could be the latent codes of a pre-trained GAN model (e.g. BigGAN [1], StyleGAN [14, 24], SWAGAN [4]), the codebook of a VQGAN model [7], Diffusion Denoising Models [3], structured representations such as a set of strokes (ClipDraw [13]) or triangles [6]”
CLIPstyler: Image Style Transfer with a Single Text Condition (2021, methodology): “Specifically, in the result of StyleGAN-NADA [6], the content images are modified to follow the text condition, but only a part of the content image can be changed, or the changes do not sufficiently reflect the semantics of the text condition.”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, methodology): “In this work, we propose an exemplar-based image editing approach that allows accurate semantic manipulation on the image content according to an exemplar image provided by users or retrieved from the database.”
Erasing Concepts from Diffusion Models (2023, unknown): “In image synthesis, a generative adversarial network (GAN) can be edited using a handful of words [14], a few sketches [46], warping gestures [47], or copy-and-paste [2].”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, methodology): “In the context of image editing, while most methods are generally limited to global edits [Chefer et al. 2022a; Crowson et al. 2022; Gal et al. 2022b; Kwon and Ye 2022], several works introduce a user-provided mask to specify the region that should be altered [Avrahami et al. 2022a; Bau et al.…”