Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

By Federico A. Galatolo, Mario Giovanni C. A. Cimino, Gigliola Vaglini. 2021

92 citations recorded by the source corpus.

In this research work we present CLIP-GLaSS, a novel zero-shot framework to generate an image (or a caption) corresponding to a given caption (or image). CLIP-GLaSS is based on the CLIP neural network, which, given an image and a descriptive caption, provides similar embeddings. Differently, CLIP-GLaSS takes a caption (or an image) as an input, and generates the image (or the caption) whose CLIP embedding is the most similar to the input one. This optimal image (or caption) is produced via a generative network, after an exploration by a genetic algorithm. Promising results are shown, based on the experimentation of the image Generators BigGAN and StyleGAN2, and of the text Generator GPT2

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “[19, 36, 32, 18] guide GANs using gradients from a CLIP model.”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, neutral): “Since CLIP provides a score of how close an image is to a caption, several works have used it to steer generative models like GANs towards a user-deﬁned text caption (Galatolo et al., 2021; Patashnik et al., 2021; Murdock, 2021; Gal et al., 2021).”
Towards Language-Free Training for Text-to-Image Generation (2021, background): “The idea of multimodal feature space also inspires some recent works on generative models [9, 10, 33, 35].”
CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders (2021, background): “CLIPDraw builds off a set of methods where rather than maximizing an activation, the objective is to minimize the distance between the produced image and a given description phrase, as deﬁned by a powerful CLIP language-image encoder (Fernando et al., 2021; Murdock, 2021; Galatolo et al., 2021).”
Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “This could be used to condition pre-trained models in a plethora of ways, for example by conditioning an image generator with a text caption using a noisy version of CLIP [42], similar to recent methods that guide GANs using text prompts [15, 40, 2].”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Deep Daze [38] optimizes the weights of a neural implicit representation network, while [37, 41, 10, 16] optimize the latent space of BigGAN [5], StyleGAN [25] or VQGAN [15].”
Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts (2021, methodology): “For example, one direction of works uses CLIP’s gradient to guide a GAN’s generator, such as guiding BigGAN [52], guiding VQGAN [43], guiding Siren [53], or a GAN with genetic algorithm-generated latent space [12].”