TediGAN: Text-Guided Diverse Face Image Generation and Manipulation

By Weihao Xia, Yujiu Yang, Jing-Hao Xue, Baoyuan Wu. 2021

465 citations recorded by the source corpus.

In this work, we propose TediGAN, a novel framework for multi-modal image generation and manipulation with textual descriptions. The proposed method consists of three components: StyleGAN inversion module, visual-linguistic similarity learning, and instance-level optimization. The inversion module maps real images to the latent space of a well-trained StyleGAN. The visual-linguistic similarity learns the text-image matching by mapping the image and text into a common embedding space. The instancelevel optimization is for identity preservation in manipulation. Our model can produce diverse and high-quality images with an unprecedented resolution at 10242. Using a control mechanism based on style-mixing, our TediGAN inherently supports image synthesis with multi-modal inputs, such as sketches or semantic labels, with or without instance guidance. To facilitate text-guided multi-modal synthesis, we propose the Multi-Modal CelebA-HQ, a large-scale dataset consisting of real face images and corresponding semantic segmentation map, sketch, and textual descriptions. Extensive experiments on the introduced dataset demonstrate the superior performance of our proposed method. Code and data are available at https://github.com/weihaox/TediGAN.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Generative Models

Citations in the corpus

Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “We consider a suite of datasets that are commonly used in literature [53, 54, 56, 59], including MSCOCO [4], CUB [47], LN-COCO [36], Multi-modal CelebA-HQ (MM CelebA-HQ) [52].”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “The disciminative model can operate on either the image space [66, 70, 27] or the latent space [80, 98].”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, neutral): “, StyleGAN [17, 37]) and attempt manipulate the style space based on textual inputs [1, 4, 14, 24, 31, 38], or rely on a pre-trained textimage matching model (e.”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “A concurrent work to ours, TediGAN [51], also uses StyleGAN for text-guided image generation and manipulation.”
GAN Inversion: A Survey (2021, background): “[131] propose a novel unified framework for both text-to-image generation and textguided image manipulation tasks by training an encoder to”
Language-Driven Image Style Transfer (2021, methodology): “Inspired by textto-image generation [19, 20, 21], previous works [22, 23, 24, 25, 26, 27, 30] perform LBIE by conditional GAN.”
Lafite2: Few-shot Text-to-Image Generation (2022, methodology): “Standard datasets are considered for downstream tasks, including MS-COCO [19], CUB [35], MM-CelebA-HQ [38], LN-COCO [24].”
Shifted Diffusion for Text-to-image Generation (2022, background): “Note that we make sure our dataset does not overlap with MS-COCO [12], CUB [27], Localized Narratives [15] and Multi-modal CelebA-HQ (MM-CelebA-HQ) [28], because we will test zero-shot or language-free performance on these downstream datasets.”
Identity-Guided Face Generation with Multi-Modal Contour Conditions (2021, background): “Semantic labels and sketches are from [20] and [21], respectively.”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, background): “Seminal works [29, 14, 46, 2] which combined these components were revolutionary, since they did not require extra manual labor, and produced”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, background): “Others have explored self-supervised approaches [?,?], the mixing of latent codes to produce local edits [11, 13, 29], and the use of contrastive language-image (CLIP) models [52] to achieve new editing capabilities [18, 48, 69].”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “Early works [1, 4, 14,43,66] leverage pretrained GAN generators [29] and text encoders [45] to progressively optimize the image according to the text prompt.”
Sound-Guided Semantic Image Manipulation (2021, methodology): “5, the proposed sound-guided image manipulation (proposed method) shows more radical results than text-guided manipulation (TediGAN [49] and StyleCLIP [34]).”
ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation (2023, background): “Albeit GAN-based [20, 35, 38] and VAE-based models [6, 11, 14, 30, 40] can synthesize images with promising quality and diversity, they still cannot match user descriptions very well.”
Pretraining is All You Need for Image-to-Image Translation (2022, background): “Considering the synergy among tasks, some research efforts [53, 60, 26, 22, 38, 6, 42] aim to learn a uniﬁed model for diverse translation tasks via multi-task training.”
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation (2021, methodology): “For the qualitative comparison of manipulation performance with other methods, we use the state-of-the-art text manipulation methods, TediGAN [44], StyleCLIP [28] and StyleGAN-NADA [16] where images”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, background): “Second, unlike image manipulation where the single-view information is enough to guide the editing [20, 44, 45], the multi-view dependency of NeRF makes the manipulation way more difﬁcult to control without the multi-view information.”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, methodology): “In the human face domain (a small data domain), we produce good results with rich diversity [46] in three image conditional synthesis tasks.”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, methodology): “Previous methods based on generative adversarial networks (GANs) [18, 15, 38, 22] have achieved some success on domain-specific datasets ( e.g ., face datasets), but they have limited applicability and generality.”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, methodology): “[71] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.”