CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders

By Kevin Frans, Lisa B. Soros, Olaf Witkowski. 2021

297 citations recorded by the source corpus.

This work presents CLIPDraw, an algorithm that synthesizes novel drawings based on natural language input. CLIPDraw does not require any training; rather a pre-trained CLIP language-image encoder is used as a metric for maximizing similarity between the given description and a generated drawing. Crucially, CLIPDraw operates over vector strokes rather than pixel images, a constraint that biases drawings towards simpler human-recognizable shapes. Results compare between CLIPDraw and other synthesis-through-optimization methods, as well as highlight various interesting behaviors of CLIPDraw, such as satisfying ambiguous text in multiple ways, reliably producing drawings in diverse artistic styles, and scaling from simple to complex visual representations as stroke count is increased. Code for experimenting with the method is available at: https://colab.research.google.com/github/kvfrans/clipdraw/blob/main/clipdraw.ipynb

Subfields: Building and Construction, Computational Mechanics, Multimodal Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Building and Construction

In Computational Mechanics

In Multimodal Learning

In Human-Computer Interaction

Citations in the corpus

CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, methodology): “BigGAN [1], StyleGAN [14, 24], SWAGAN [4]), the codebook of a VQGAN model [7], Diffusion Denoising Models [3], structured representations such as a set of strokes (ClipDraw [13]) or triangles [6], or a SIREN network [2, 5] that maps spatial coordinates to pixels.”
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance (2022, methodology): “Researchers using our framework have introduced a number of additional components, ranging from using ensembles [6], to using Bézier curves for latent representations [13, 5], to using perturbations to make the results more robust to adversaries [25].”
StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis (2021, methodology): “The baseline is formed by using CLIPDraw [1] to convert the input text into an image, then performing style transfer [2].”
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation (2021, methodology): “Recently, pre-trained CLIP has been used for several zero-shot downstream applications [15, 17, 25, 40, 44, 53].”
Long-CLIP: Unlocking the Long-Text Capability of CLIP (2024, methodology): “As CLIP has strong zero-shot generalization abilities, it has been successfully utilized in detection [7,13], segmentation [12,31], video understanding [16,30] and most commonly, image generating [4,6,21,28].”
Text2Mesh: Text-Driven Neural Stylization for Meshes (2021, background): “In [10] random 2D augmentations are necessary to generate meaningful CLIP-guided drawings.”
Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? (2022, background): “, 2021) representation models that have widely impacted downstream applications in vision (Tian et al., 2020; Conde and Turgutlu, 2021; Patashnik et al., 2021; Frans et al., 2021) and NLP (Lin et al.”
Text2LIVE: Text-Driven Layered Image and Video Editing (2022, methodology): “Several works [11,14,21,28] take a test-time optimization approach and leverage CLIP without using a pre-trained generator.”
Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts (2021, methodology): “Usually, to make such kind of combination work ideally, more studies are required, which warrant a manuscript itself like [11, 43].”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, background): “This expressive representation enables high quality image generation and editing, controlled by natural language [Patashnik et al. 2021; Gal et al. 2021; Frans et al. 2021].”