MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders

By Xuezhe Ma, Chunting Zhou, Eduard H. Hovy. 2019

41 citations recorded by the source corpus.

Variational Autoencoder (VAE), a simple and effective deep generative model, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations. However, recent studies demonstrate that, when equipped with expressive generative distributions (aka. decoders), VAE suffers from learning uninformative latent representations with the observation called KL Varnishing, in which case VAE collapses into an unconditional generative model. In this work, we introduce mutual posterior-divergence regularization, a novel regularization that is able to control the geometry of the latent space to accomplish meaningful representation learning, while achieving comparable or superior capability of density estimation. Experiments on three image benchmark datasets demonstrate that, when equipped with powerful decoders, our model performs well both on density estimation and representation learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, background): “[92] Xuezhe Ma, Chunting Zhou, and Eduard Hovy.”
Deep Mixture Generative Autoencoders (2021, methodology): “[34] introduced a new regularization term in the”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “latents to capture global structure [28]; a mutual information term can be added to the loss to encourage high correlation between x and z [259]; encouraging the posterior to be diverse by controlling its geometry to evenly covering the data space, redundancy is reduced and latents are encouraged to”
MaCow: Masked Convolutional Generative Flow (2019, methodology): “The (Reed et al., 2017) – 3.70 PixelRNN (Oord et al., 2016) 3.00 3.63 Gated PixelCNN (van den Oord et al., 2016) 3.03 3.57 MAE (Ma et al., 2019) 2.95 – PixelCNN++ (Salimans et al., 2017) 2.92 – PixelSNAIL (Chen et al., 2017) 2.85 3.52 SPN (Menick and Kalchbrenner, 2019) – 3.52”
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019, unknown): “Several other papers have studied alternative formulations of the VAE objective [34, 13, 2, 29, 41].”