IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis

By Huaibo Huang, Zhihang Li, Ran He, Zhenan Sun, Tien-Ping Tan. 2018

299 citations recorded by the source corpus.

We present a novel introspective variational autoencoder (IntroVAE) model for synthesizing high-resolution photographic images. IntroVAE is capable of self-evaluating the quality of its generated samples and improving itself accordingly. Its inference and generator models are jointly trained in an introspective way. On one hand, the generator is required to reconstruct the input images from the noisy outputs of the inference model as normal VAEs. On the other hand, the inference model is encouraged to classify between the generated and real samples while the generator tries to fool it as GANs. These two famous generative frameworks are integrated in a simple yet efficient single-stream architecture that can be trained in a single stage. IntroVAE preserves the advantages of VAEs, such as stable training and nice latent manifold. Unlike most other hybrid models of VAEs and GANs, IntroVAE requires no extra discriminators, because the inference model itself serves as a discriminator to distinguish between the generated and real samples. Experiments demonstrate that our method produces high-resolution photo-realistic images (e.g., CELEBA images at \(1024^{2}\)), which are comparable to or better than the state-of-the-art GANs.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Dual Contradistinctive Generative Autoencoder (2020, positive): “IntroVAE [29] adds a loop from the output back to the input and is able to attain image quality that is on par with some modern GANs in some aspects.”
Adversarial Latent Autoencoders (2020, background): “PIONEER [15] and IntroVAE [20] followed this line, with the latter providing the best generation results in this category.”
COCO-GAN: Generation by Parts via Conditional Coordinating (2019, background): “Recent studies in high-resolution image generation [13, 17, 12] have gained lots of success; however, a shared conundrum among these existing approaches is the computation being memory hungry.”
Image Generation From Small Datasets via Batch Statistics Adaptation (2019, background): “In recent years, image generation using deep generative models has rapidly developed, and some state-of-the-art methods can generate images that cannot be distinguished from real data [14, 6, 24, 33, 11, 20, 36, 8, 18].”
An Interpretable Generative Model for Handwritten Digits Synthesis (2019, methodology): “[16] proposed an introspective variational autoencoder (IntroVAE) model to synthesize high-resolution photographic images.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “Other proposed improvements include combining with adversarial training [97],”
Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods (2019, methodology): “…Denoising Auto-Encoders (DAE) [12], Contractive Auto-Encoders (CAE) [13] Generative Adversarial Nets (GAN) GAN [14], BiGAN [15], ALI [16], IntroAVE [17], VEEGAN [18] Disentangled Representations: InfoGAN [19], β -VAE [20], FactorVAE [21] More Generative Models Flow-based Generative Models: GLOW…”
Towards Photographic Image Manipulation with Balanced Growing of Generative Autoencoders (2019, methodology): “We seek to balance the training in higher resolutions—a common challenge for generative models (see e.g . discussion in [16]).”
From Variational to Deterministic Autoencoders (2019, background): “Much of the recent literature has focused on applying VAEs on image generation tasks [19, 23, 33] and devising new encoder–decoder architectures [25, 56].”
Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition (2019, methodology): “The more recent IntroVAE [13] and adversarial encoder generator networks [14] have subsequently come to the realization that this doesn’t necessarily require the auxiliary real-fake discriminator, as the VAE itself already provides strong means for discrimination, namely its probabilistic encoder.”
A survey on face data augmentation for the training of deep neural networks (2019, methodology): “[107] proposed an introspective variational autoencoder (IntroVAE) model for synthesizing high-resolution photorealistic images by selfevaluating the quality of the generated samples during the training process.”
Face identity disentanglement via latent space mapping (2020, methodology): “Some methods train an encoder to map images to the latent space in conjunction to training the generator (e.g., [Huang et al. 2018; Pidhorskyi et al. 2020]).”
PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Driven Adaptive Prior (2021, methodology): “Inspired by the cycle consistency objective commonly used in other classes of generative models [8, 41], we further propose applying the consistency objective of the estimated noisy signal from the neural network on the entire forward diffusion process.”