Paint by Word

By David Bau, Alex Andonian, Audrey Cui, Yeonhwa Park, Ali Jahanian, Aude Oliva, Antonio Torralba. 2021

138 citations recorded by the source corpus.

We investigate the problem of zero-shot semantic image painting. Instead of painting modifications into an image using only concrete colors or a finite set of semantic concepts, we ask how to create semantic paint based on open full-text descriptions: our goal is to be able to point to a location in a synthesized image and apply an arbitrary new concept such as"rustic"or"opulent"or"happy dog."To do this, our method combines a state-of-the art generative model of realistic images with a state-of-the-art text-image semantic similarity network. We find that, to make large changes, it is important to use non-gradient methods to explore latent space, and it is important to relax the computations of the GAN to target changes to a specific region. We conduct user studies to compare our methods to several baselines.

Subfields: Cognitive Neuroscience, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, neutral): “…r s ( ﬁ n e - t un e d ) O u r s ( i m p li c it ) “pink yarn ball” “red dog collar” “dog bone” “pizza” “golden necklace” “blooming tree” “tie with black “blue short pants” and yellow stripes” 2) PaintByWord++ (Bau et al., 2021; Avrahami et al., 2021), (3) Blended Diffusion (Avrahami et al., 2021).”
StyleGAN-NADA (2021, methodology): “The representations learned by CLIP have been used in guiding a variety of tasks, including image synthesis [12,32] and manipulation [7,37].”
GAN Inversion: A Survey (2021, methodology): “On the other hand, a few methods [135], [143], [144] first obtain the latent code of a given image and find the target latent code of desired attributes with the guidance of some powerful pretrained language models, e.”
ClipCap: CLIP Prefix for Image Captioning (2021, background): “Many works have already used CLIP successfully for computer vision tasks that require the understanding of some auxiliary text, such as generating or editing an image based on a natural language condition [5, 14, 28].”
Alias-Free Generative Adversarial Networks (2021, background): “While several ways of controlling the generative process have been found [8, 28, 10, 41, 24, 2, 7, 46, 6], the foundations of the synthesis process remain only partially understood.”
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion (2021, background): “Compared with Paint By Word[2], NÜWA shows strong manipulation ability, generating highquality text-consistent results while not changing other parts of the image.”
Blended Latent Diffusion (2022, unknown): “Local text-guided image manipulation: Paint By Word [Bau et al. 2021] was first to address the problem of zero-shot local textguided image manipulation by combining BigGAN / StyleGAN with CLIP and editing only the part of the feature map that corresponds to the input mask.”
Text2LIVE: Text-Driven Layered Image and Video Editing (2022, methodology): “Instead of directly training a text-to-image generator, a recent surge of meth-ods leverage a pre-trained generator, and use a pre-trained CLIP [35] to guide the generation process by text [3,12,25,33].”
High-fidelity GAN Inversion with Padding Space (2022, background): “Existing studies on GAN interpretation have affirmed that, pre-trained GAN models own great potential in a range of downstream applications, such as object classification [9, 45], semantic segmentation [49], video generation [23], and image editing [4, 11, 13, 20, 28, 31, 38, 39, 42, 46, 50].”
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting (2022, unknown): “There has been much recent work on text-guided image inpainting [1, 3, 5, 7, 19, 27, 31, 33].”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Compared with text-conditioned inpainting methods [3,6,51] that apply a single concept to an image region, “paint-with-words” can generate the whole image containing multiple concepts in a single pass from scratch, without the need to start from an input image.”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “Early works [1, 4, 14,42,66] leverage pretrained GAN generators [29] and text encoders [44] to progressively optimize the image according to the text prompt.”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, background): “Bau et al. [7] further demonstrated how to use masks provided by the user, to localize the text-based editing and restrict the change to a speciﬁc spatial region.”
Creating, Using, Misusing, and Detecting Deep Fakes (2022, background): “Even more powerfully, this stylizing can be driven by text-based descriptions alone (Razavi, Van den Oord, and Vinyals 2019; Radford et al. 2021; Bau et al. 2021).”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, methodology): “Text-driven image manipulation has recently achieved significant progress using GANs [9,22,27–29] combined with image-text representations such as CLIP [50], yielding realistic manipulations using text [2, 7, 21, 41, 46, 68].”
DiffEdit: Diffusion-based semantic image editing with mask guidance (2022, unknown): “The gradient of a CLIP score can also be used to match a given text query inside a mask, as in Paint by Word (Bau et al., 2021), local CLIP-guided diffusion (Crowson, 2021), or blended diffusion (Avrahami et al., 2022).”