Lagging Inference Networks and Posterior Collapse in Variational Autoencoders

By Junxian He, Daniel M. Spokoyny, Graham Neubig, Taylor Berg-Kirkpatrick. 2019

307 citations recorded by the source corpus.

The variational autoencoder (VAE) is a popular combination of deep latent variable model and accompanying variational learning technique. By using a neural inference network to approximate the model's posterior on latent variables, VAEs efficiently parameterize a lower bound on marginal data likelihood that can be optimized directly via gradient methods. In practice, however, VAE training often results in a degenerate local optimum known as "posterior collapse" where the model learns to ignore the latent variable and the approximate posterior mimics the prior. In this paper, we investigate posterior collapse from the perspective of training dynamics. We find that during the initial stages of training the inference network fails to approximate the model's true posterior, which is a moving target. As a result, the model is encouraged to ignore the latent encoding and posterior collapse occurs. Based on this observation, we propose an extremely simple modification to VAE training to reduce inference lag: depending on the model's current mutual information between latent variable and observation, we aggressively optimize the inference network before performing each model update. Despite introducing neither new model components nor significant complexity over basic VAE, our approach is able to avoid the problem of collapse that has plagued a large amount of previous work. Empirically, our approach outperforms strong autoregressive baselines on text and image benchmarks in terms of held-out likelihood, and is competitive with more complex techniques for avoiding collapse while being substantially faster.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

CogView: Mastering Text-to-Image Generation via Transformers (2021, background): “However, it leads to posterior collapse [23], meaning that q ( z | x i ; φ ) sometimes collapses towards the prior.”
Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction (2019, methodology): “As methods to mitigate these effects are still areas of active research (see references in He et al. (2019) and Tomczak & Welling (2018)), we also train the RNF without the KL regularisation terms – which can be interpreted as placing a ﬂat prior over the latent state and removing constants from…”
Implicit Generation and Generalization in Energy-Based Models (2019, background): “There is no need to tune training processes for separate networks to make sure they are balanced (for example, (He et al., 2019) point out unbalanced training can result in posterior collapse in VAEs or poor performance in GANs (Kurach et al., 2018)).”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “A vast body of work has tried to alleviate the mode collapse problem in GANs [11, 12, 65, 4, 25, 16, 47] and the posterior collapse problem in VAEs [76, 66, 36, 8, 48, 29].”
Residual Energy-Based Models for Text (2021, background): “…auto-encoders have been investigated for representing and generating text (Bowman et al., 2016; Zhao et al., 2018; He et al., 2019), but most have not shown signiﬁcant improvements in terms of perplexity, with the notable exception of He et al. (2019) which outperforms an LSTM-based language model.”
PortaSpeech: Portable and High-Quality Generative Text-to-Speech (2021, methodology): “However, its generated mel-spectrograms are very blurry and over-smoothing, resulting in unnatural sounds, due to the posterior collapse [1, 6] and the reconstruction loss term used in BVAE-TTS, which has independency assumption of generated frequency bins given latent variables.”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, background): “EBMs allow more flexible architectures [9, 11] with simpler compositionality [15, 41] compared to other explicit generative models [46, 16], and have better stability and mode coverage in training [30, 31, 71] compared to implicit generative models [10].”
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing (2019, methodology): “Concurrent with our work, He et al. (2019) proposed to update encoder multiple times to achieve better latent code before updating decoder.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “Quality can be improved using higher capacity functions, however these can be harder to balance, leading to posterior collapse [69], [228], where even in the asymptotic regime the true posterior still cannot be recovered [166].”
Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (2020, methodology): “( iii ) Aggressive Training (He et al., 2019).”
A Bilingual Generative Transformer for Semantic Sentence Embedding (2019, methodology): “We trained two LSTM VAE models following the settings and aggressive training strategy in (He et al., 2019), where one LSTM model used the Hidden strategy and the other used the Hidden + Logit strategy.”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, methodology): “Although cVAE is easy to train and sample due to its simple architecture and one-step sampling procedure, it suffers from the posterior collapse problem [12, 17, 26, 64, 69, 84, 93]; the learned latent variable is ignored by a strong decoder, leading to limited generation diversity from these collap”
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2023, methodology): “For baselines, we compare against Lag VAE (He et al., 2019), CGMH (Miao et al., 2019), UPSA (Liu et al., 2020), BT (Wieting and Gimpel, 2018), Corruption (Hegde and Patil, 2020), Con-RPG (Meng et al., 2021), and MCPG (Chen et al., 2022b).”
On Variational Learning of Controllable Representations for Text without Supervision (2019, methodology): “For aggressive training, we follow He et al. (2019) to aggressively train the encoding network.”
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019, unknown): “[17] motivate posterior collapse through an investigation of the learning dynamics of deep VAEs.”
Mixture Content Selection for Diverse Sequence Generation (2019, unknown): “Dieng et al., 2018; Xu and Durrett, 2018; He et al., 2019; Razavi et al., 2019).”
Implicit Generation and Modeling with Energy Based Models (2019, background): “Separate networks are not tuned to ensure balance (for example, [He et al., 2019] point out unbalanced training can result in posterior collapse in VAEs or poor performance in GANs [Kurach et al.”