Amortized Inference Regularization

By Rui Shu, Hung Hai Bui, Shengjia Zhao, Mykel J. Kochenderfer, Stefano Ermon. 2018

89 citations recorded by the source corpus.

The variational autoencoder (VAE) is a popular model for density estimation and representation learning. Canonically, the variational principle suggests to prefer an expressive inference model so that the variational approximation is accurate. However, it is often overlooked that an overly-expressive inference model can be detrimental to the test set performance of both the amortized posterior approximator and, more importantly, the generative density estimator. In this paper, we leverage the fact that VAEs rely on amortized inference and propose techniques for amortized inference regularization (AIR) that control the smoothness of the inference model. We demonstrate that, by applying AIR, it is possible to improve VAE generalization on both inference and generative performance. Our paper challenges the belief that amortized inference is simply a mechanism for approximating maximum likelihood training and illustrates that regularization of the amortization family provides a new direction for understanding and improving generalization in VAEs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Probabilistic Model-Agnostic Meta-Learning (2018, methodology): “We can further avoid storing a separate variational distribution qi(φi|θ) and qi(θ) for each task Ti by employing an amortized variational inference technique [22, 21, 36], where we instead set qi(φi|θ) = qψ(φi|θ,x tr i ,y tr i ,x test i ,y test i ), where qψ is defined by some function approximator”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “Shu et al. (2018)), and work better in practice.”
InfoVAE: Balancing Learning and Inference in Variational Autoencoders (2019, background): “2) Overfitting: Because pD is an empirical (finite) distribution in practice, matching it too closely can lead to poor generalization (Shu et al. 2018).”
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing (2019, background): “|| It is determined by the amortized inference regularization (implied by the encoder’s expressivity) (Shu et al., 2018), which further affects the performance of the generative density estimator.”
Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection (2019, background): “It might also be interpreted as regularizing the encoder [63], or as increasing its flexibility [70].”
Meta-Amortized Variational Inference and Learning (2019, background): “Then, Lφ,θ(pi,x) = −DKL(pi(x)qφ(z|x, pi)||p(z)pθi(x|z)) = −DKL(pi(x)||pθi(x)) − Ex∼pi [DKL(qφ(z|x, pi)||pθi(z|x))] As in (Shu et al., 2018), this MetaELBO has a maximum likelihood term and a regularization term but for each distribution pi(x), thereby encouraging the meta-amortized inference model t”
Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis (2018, background): “(See [77], [78] for in-depth explanations.)”