Bridging CLIP and StyleGAN through Latent Alignment for Image Editing

By Wanfeng Zheng, Qiang Li, Xiaoyan Guo, Pengfei Wan, Zhong-ming Wang. 2022

16 citations recorded by the source corpus.

Text-driven image manipulation is developed since the vision-language model (CLIP) has been proposed. Previous work has adopted CLIP to design a text-image consistency-based objective to address this issue. However, these methods require either test-time optimization or image feature cluster analysis for single-mode manipulation direction. In this paper, we manage to achieve inference-time optimization-free diverse manipulation direction mining by bridging CLIP and StyleGAN through Latent Alignment (CSLA). More specifically, our efforts consist of three parts: 1) a data-free training strategy to train latent mappers to bridge the latent space of CLIP and StyleGAN; 2) for more precise mapping, temporal relative consistency is proposed to address the knowledge distribution bias problem among different latent spaces; 3) to refine the mapped latent in s space, adaptive style mixing is also proposed. With this mapping scheme, we can achieve GAN inversion, text-to-image generation and text-driven image manipulation. Qualitative and quantitative comparisons are made to demonstrate the effectiveness of our method.

Subfields: Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Multimodal Learning

Citations in the corpus

InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Recent models have leveraged CLIP [47] embeddings to guide image editing using text [5, 9, 14, 28, 31, 41, 44, 70].”