TediGAN: Text-Guided Diverse Image Generation and Manipulation

By Weihao Xia, Yujiu Yang, Jing-Hao Xue, Baoyuan Wu. 2020

24 citations recorded by the source corpus.

In this work, we propose TediGAN, a novel framework for multi-modal image generation and manipulation with textual descriptions. The proposed method consists of three components: StyleGAN inversion module, visual-linguistic similarity learning, and instance-level optimization. The inversion module is to train an image encoder to map real images to the latent space of a well-trained StyleGAN. The visual-linguistic similarity is to learn the text-image matching by mapping the image and text into a common embedding space. The instance-level optimization is for identity preservation in manipulation. Our model can provide the lowest effect guarantee, and produce diverse and high-quality images with an unprecedented resolution at 1024. Using a control mechanism based on style-mixing, our TediGAN inherently supports image synthesis with multi-modal inputs, such as sketches or semantic labels with or without instance (text or real image) guidance. To facilitate text-guided multi-modal synthesis, we propose the Multi-Modal CelebA-HQ, a large-scale dataset consisting of real face images and corresponding semantic segmentation map, sketch, and textual descriptions. Extensive experiments on the introduced dataset demonstrate the superior performance of our proposed method. Code and data are available at https://github.com/weihaox/TediGAN.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Generative Models

In Multimodal Learning

Citations in the corpus

GAN Inversion: A Survey (2021, background): “In addition to common applications in image processing, in the last few months, GAN inversion techniques have been widely introduced to many other tasks, such as 3D reconstruction [139], [140], image understanding [141], [142], multimodal learning [133], [134], [135], [143], [144], and medical imagi”
UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis (2021, methodology): “First, we compare our UFC-BERT with GAN-based text-to-image models AttnGAN [62], ControlGAN [30], DF-GAN [54], DM-GAN [69] and TediGAN [61] on the Multi-Modal CelebA-HQ dataset.”
M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers (2021, methodology): “Existing works [63, 26, 62] hence typically design separate methods customized for each control modality.”
StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulation (2021, methodology): “One of the recent works that uses image-text matching methods is TediGAN [35], which inverts real images using the inversion module of StyleGAN and then learns the correspondences between visual and linguistic attributes.”
Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, methodology): “Compared with TediGAN-A, which is the initial strategy proposed in the conference version [19], the newly proposed TediGAN-B can further handle image creation from openworld images or texts and region-of-interest manipulation, which will be discussed in Section 4.”
Cycle-Consistent Inverse GAN for Text-to-Image Synthesis (2021, methodology): “Another reason is that the latent space W of StyleGAN is still entangled with some semantic attributes [33], which means that when we change part of the attributes, the rest attributes will also be affected.”