Improved Variational Inference with Inverse Autoregressive Flow

By Diederik P. Kingma, Tim Salimans, Max Welling. 2016

2,023 citations recorded by the source corpus.

The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network. In experiments, we show that IAF significantly improves upon diagonal Gaussian approximate posteriors. In addition, we demonstrate that a novel type of variational autoencoder, coupled with IAF, is competitive with neural autoregressive models in terms of attained log-likelihood on natural images, while allowing significantly faster synthesis.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, methodology): “) Similar to the discretized continuous distributions used in VAE decoders and autoregressive models [34, 52], our choice here ensures that the variational bound is a lossless codelength of discrete data, without need of adding noise to the data or incorporating the Jacobian of the scaling operation”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, methodology): “A further developed version of this model is presented in [17], where the architecture is explained in detail.”
Jukebox: A Generative Model for Music (2020, methodology): “Autoregressive flows (Papamakarios et al., 2017; Kingma et al., 2016) combine the two ideas to achieve faster density estimation or data generation.”
Neural Discrete Representation Learning (2017, methodology): “Extensions include autoregressive prior and posterior models [14], normalising flows [31, 10], and inverse autoregressive posteriors [22].”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, background): “Inverse autoregressive flows (IAFs) [15] represent a kind of dual formulation of deep autoregressive modelling, in which sampling can be performed in parallel, while the inference procedure required for likelihood estimation is sequential and slow.”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “Recently, ﬂow-based models (henceforth simply called ﬂow models ) have gained popularity as a type of deep generative model [7, 8, 32, 17, 3, 37, 68, 10] and for use in variational inference [34, 59, 33].”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “…models with latent variables following the inﬂuential work by Kingma and Welling [37], Rezende et al. [59], and models optimized with this bound have been successfully used to model data such as natural images [58, 38, 11, 26], speech and music time-series [12, 23, 39], and video [2, 28, 17].”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “58 IAFVAE L 1 [23] ≤ 3 .”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “One subclass of such methods, based on variational inference, allows for efﬁcient approximate inference and sampling, but does not admit exact log likelihood computation (Kingma & Welling, 2013; Rezende et al., 2014; Kingma et al., 2016).”
Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, methodology): “The student networks underlying both Parallel WaveNet and Clarinet use Inverse Auto-regressive Flows (IAF) [10].”
FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, methodology): “By using an autoregressive model and specifying an ordering of the dimensions, the Jacobian of f is enforced to be lower triangular (Kingma et al., 2016; Oliva et al., 2018).”
Glow: Generative Flow with Invertible 1x1 Convolutions (2018, background): “Variational autoencoders have the advantage of parallelizability of training and synthesis, but can be comparatively challenging to optimize (Kingma et al., 2016).”
Neural Ordinary Differential Equations (2018, neutral): “Recent work explores the tradeoff between the expressiveness of normalizing ﬂ ow layers and computational cost (Kingma et al., 2016; Tomczak and Welling, 2016; Berg et al., 2018).”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, background): “, 1995) or variants thereof (Kingma & Welling, 2013; Rezende et al., 2014; de Freitas et al., 2001; Gregor et al., 2015b;a; Tran et al., 2015; Kingma et al., 2016) trained with approximate inference.”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, methodology): “4.40 DRAW (Gregor et al., 2015) 4.13 ResNet VAE with IAF (Kingma et al., 2016) 3.11 Table 1: Comparisons on CIFAR10, test set average number of bits/data dimension(lower is better) 6 Discussion”
Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, neutral): “VAEs are easy to train but tend to produce blurry results due to restrictions in the model, although recent work is improving this (Kingma et al., 2016).”
A-NICE-MC: Adversarial Training for MCMC (2017, background): “A flow model [12, 26, 27] defines a generative model for x ∈ R through a bijection f : h → x, where h ∈ R have the same number of dimensions as x with a fixed prior pH(h) (typically a factored Gaussian).”
Density estimation using Real NVP (2016, background): “Recent work in this line of research has taken advantage of recent advances in recurrent networks [51], in particular long-short term memory [26], and residual networks [25, 24] in order to learn state-of-the-art generative image models [61, 46] and language models [32].”