CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training

By Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, Gang Hua. 2017

266 citations recorded by the source corpus.

We present variational generative adversarial networks, a general learning framework that combines a variational auto-encoder with a generative adversarial network, for synthesizing images in fine-grained categories, such as faces of a specific person or objects in a category. Our approach models an image as a composition of label and latent attributes in a probabilistic model. By varying the fine-grained category label fed into the resulting generative model, we can generate images in a specific category with randomly drawn values on a latent attribute vector. Our approach has two novel aspects. First, we adopt a cross entropy loss for the discriminative and classifier network, but a mean discrepancy objective for the generative network. This kind of asymmetric loss function makes the GAN training more stable. Second, we adopt an encoder network to learn the relationship between the latent space and the real image space, and use pairwise feature matching to keep the structure of generated images. We experiment with natural images of faces, flowers, and birds, and demonstrate that the proposed models are capable of generating realistic and diverse samples with fine-grained category labels. We further show that our models can be applied to other tasks, such as image inpainting, super-resolution, and data augmentation for training better face recognition models.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age (2019, background): “To date, numerous large scale face image datasets [22, 31, 13, 69, 37, 24, 41, 68, 15, 27, 46, 8, 39] have been proposed and fostered research and development for automated face detection [35, 21], alignment [66, 44], recognition [57, 49], generation [67, 5, 26, 58], modification [3, 32, 19], and at”
Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations (2018, background): “…of a generally useful latent space (disentanglement, linearity, sparsity, ...) and explore new objective functions that can be leveraged for unsupervised representation learning, such as various representation losses [8, 9], GAN-inspired discriminator networks [10] and predictive capacity [11, 12].”
Learning to Dress 3D People in Generative Clothing (2019, methodology): “4), under the framework of a VAE-GAN [7, 29], using graph convolutions [11] and mesh sampling [41] as the backbone layers.”
Diversity-Generated Image Inpainting with Style Extraction (2019, methodology): “The encoder in CVAE-GAN [14] takes the extracted features of the ground truth image as the input of the generator directly.”
Semantically Multi-Modal Image Synthesis (2020, unknown): “Previous works utilized different strategies for the task: adopting the idea of variational auto-encoder [36, 56, 2, 11], introducing noise while training [19], building multiple subnetworks [10] and including instance-level feature embeddings [41], etc.”