Semi-Amortized Variational Autoencoders

By Yoon Kim, Sam Wiseman, Andrew C. Miller, David A. Sontag, Alexander M. Rush. 2018

266 citations recorded by the source corpus.

Amortized variational inference (AVI) replaces instance-specific local inference with a global inference network. While AVI has enabled efficient training of deep generative models such as variational autoencoders (VAE), recent empirical work suggests that inference networks can produce suboptimal variational parameters. We propose a hybrid approach, to use AVI to initialize the variational parameters and run stochastic variational inference (SVI) to refine them. Crucially, the local SVI procedure is itself differentiable, so the inference network and generative model can be trained end-to-end with gradient-based optimization. This semi-amortized approach enables the use of rich generative models without experiencing the posterior-collapse phenomenon common in training VAEs for problems like text generation. Experiments show this approach outperforms strong autoregressive and variational baselines on standard text and image datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Meta-Learning Probabilistic Inference for Prediction (2018, background): “(cid:12)(cid:12) This is an example of semi-amortized inference (Kim et al., 2018b), as the only shared inference parameters are the initialization and learning rate, and optimization is required for each task (albeit only for one step).”
Accelerating Protein Design Using Autoregressive Generative Models (2019, neutral): “Though others have reported difﬁculty training these models (Killoran et al., 2017), new machine learning techniques may make these models practical in the future (Kim et al., 2018; Yang et al., 2017; van den Oord & Vinyals, 2017).”
Preventing Posterior Collapse with delta-VAEs (2019, background): “…has been frequently observed in prior work and has been referred to as optimisation challenges of VAEs by Bowman et al. (2015), the information preference property by Chen et al. (2016), and the posterior collapse problems by several others (e.g., van den Oord et al. (2017); Kim et al. (2018)).”
Iterative Amortized Inference (2018, methodology): “For inference optimization, previous works have combined standard inference models with gradient updates (Hjelm et al., 2016; Krishnan et al., 2018; Kim et al., 2018), however, these works do not learn to iteratively optimize.”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, methodology): “Then we follow the synthetic data generation procedure in Kim et al. (2018), where we sample data points from an one-layer LSTM conditioned on latent variables.”
Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (2020, background): “Language VAEs have inspired new applications in NLP, via exploiting many interesting properties of the model’s latent space (Bowman et al., 2016; Kim et al., 2018b).”
Action and Perception as Divergence Minimization (2020, background): “It can also compute beliefs for new inputs without further optimization, although optimization can refine the belief (Kim et al., 2018).”
Latent Normalizing Flows for Discrete Sequences (2019, methodology): “, 2017), or the variational inference procedure (Kim et al., 2018), though in all cases the model still relies heavily on the discrete decoder.”
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019, unknown): “A related line of research studies issues arising from approximate inference causing a mismatch between the variational distribution and true posterior [12, 22, 19].”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “A vast body of work has tried to alleviate the mode collapse problem in GANs [11, 12, 65, 4, 25, 16, 47] and the posterior collapse problem in VAEs [76, 66, 36, 8, 48, 29].”
Consistency Regularization for Variational Auto-Encoders (2021, background): “Researchers have looked at ways to improve the quality of the latent representations of vaes, often tackling the so-called latent variable collapse problem—in which the approximate posterior distribution induced by the encoder collapses to the prior over the latent variables (Bowman et al., 2015; Ki”
Latent Alignment and Variational Attention (2018, methodology): “A hybrid method such as semi-amortized inference [39, 34] might be a potential future direction worth exploring.”
Learning Latent Space Energy-Based Prior Model (2020, methodology): “We compare our model to related baselines, SA-VAE [27], FB-VAE [35], and ARAE [63].”
Avoiding Latent Variable Collapse With Generative Skip Models (2018, methodology): “We also study the semi-amortized variational autoencoder, sa-vae [Kim et al., 2018], which proposes a different optimization-based strategy for targeting the latent variable collapse issue when training vaes for text.”
COIN: COmpression with Implicit Neural representations (2021, background): “In a similar vein to work in the latent variable model literature [14, 19, 16, 22], several works [5, 12, 39] attempt to close the amortization gap [8] by performing iterative gradient-based optimization steps on top of the use of amortized inference networks.”
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing (2019, methodology): “Semi-amortized training (Kim et al., 2018) was proposed to perform stochastic variational inference (SVI) (Hoffman et al., 2013) on top of the amortized inference in VAE. high, while our monotonic schedule does It shares a similar motivation with the proposed approach, in that better latent codes…”
VoxelMorph: A Learning Framework for Deformable Medical Image Registration (2018, background): “parameters, which in other domains has been referred to as amortization [12]–[15].”
Maximizing Mutual Information for Tacotron (2019, methodology): “There is a frequently observed problem in variational autoencoder (VAE) [22] training called optimization challenge [4] or posterior collapse [20, 33].”