Variational Inference with Normalizing Flows
By Danilo Jimenez Rezende, Shakir Mohamed. 2015
5,033 citations recorded by the source corpus.
The choice of approximate posterior distribution is one of the core problems in variational inference. Most applications of variational inference employ simple families of posterior approximations in order to allow for efficient inference, focusing on mean-field or other simple structured approximations. This restriction has a significant impact on the quality of inferences made using variational methods. We introduce a new approach for specifying flexible, arbitrarily complex and scalable approximate posterior distributions. Our approximations are distributions constructed through a normalizing flow, whereby a simple initial density is transformed into a more complex one by applying a sequence of invertible transformations until a desired level of complexity is attained. We use this view of normalizing flows to develop categories of finite and infinitesimal flows and provide a unified view of approaches for constructing rich posterior approximations. We demonstrate that the theoretical advantages of having posteriors that better match the true posterior, combined with the scalability of amortized variational approaches, provides a clear improvement in performance and applicability of variational inference.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Generative Models, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Quoc V. Le
- Ruslan Salakhutdinov
- Jianfeng Gao
- Percy Liang
- Kyunghyun Cho
In Statistical and Nonlinear Physics
- Stefano Ermon
- Arash Vahdat
- Yang Song
- Yoshua Bengio
- Jun Zhu
- Diederik P. Kingma
- Jiaming Song
- Jonathan Ho
In Generative Models
- Stefano Ermon
- Arash Vahdat
- Yoshua Bengio
- Diederik P. Kingma
- Jiaming Song
- Aaron C. Courville
- Jan Kautz
- Jonathan Ho
In Probabilistic & Bayesian Methods
- Stefano Ermon
- Yoshua Bengio
- Max Welling
- David M. Blei
- Diederik P. Kingma
- Richard E. Turner
- Jun Zhu
- Yang Song
Citations in the corpus
- Denoising Diffusion Probabilistic Models (2020, background): “While diffusion models might resemble flows [9, 46, 10, 32, 5, 16, 23] and VAEs [33, 47, 37], diffusion models are designed so that q has no parameters and the top-level latent xT has nearly zero mutual information with the data x0.”
- Jukebox: A Generative Model for Music (2020, background): “…of probability to factorize the joint distribution of data into a product of simpler distributions, and flow-based models (Dinh et al., 2015; 2017; Rezende & Mohamed, 2015; Kingma & Dhari-wal, 2018) learn a series of invertible transformations that maps the data distribution with a simpler one…”
- Neural Discrete Representation Learning (2017, methodology): “Extensions include autoregressive prior and posterior models [14], normalising flows [31, 10], and inverse autoregressive posteriors [22].”
- Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, background): “IAFs are a special type of normalising flow [3, 22, 4] which model a multivariate distribution pX(x) as an explicit invertible non-linear transformation f of a simple tractable distribution pZ(z) (such as an isotropic Gaussian distribution).”
- Stochastic Normalizing Flows (2020, methodology): “The second column is the mean value of − log ˆ P D ( s ) on the test data set (related to Fig.4c in (Rezende & Mohamed, 2015)).”
- Flow Contrastive Estimation of Energy-Based Models (2019, background): “Recently, flow-based models (henceforth simply called flow models ) have gained popularity as a type of deep generative model [7, 8, 32, 17, 3, 37, 68, 10] and for use in variational inference [34, 59, 33].”
- Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “…models with latent variables following the influential work by Kingma and Welling [37], Rezende et al. [59], and models optimized with this bound have been successfully used to model data such as natural images [58, 38, 11, 26], speech and music time-series [12, 23, 39], and video [2, 28, 17].”
- Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “We distinguish two main types of generative models: likelihood based models, which include VAEs [16, 31], flow based [9, 30, 10, 17] and autoregressive models [20, 39]; and implicit generative ∗Equal contributions.”
- BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “Here, the complementary research into learning a covariance structure through a framework of normalizing flows [39, 52, 23] and the stacking of a hierarchy of latent variables [4, 37, 31, 50] have shown promising results.”
- Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, background): “The key idea in the de-sign of a flow model is to form f by stacking individual simple invertible transformations (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; Rezende & Mohamed, 2015; Kingma et al., 2016; Louizos & Welling, 2017).”
- Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, background): “The sequence of transformations is also referred to as a normalizing flow [13].”
- FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, background): “For example, each layer of the original normalizing flows model of Rezende & Mohamed (2015) is a one-layer neural network with only a single hidden unit.”
- Glow: Generative Flow with Invertible 1x1 Convolutions (2018, background): “While it may look intimidating, its value can be surprisingly simple to compute for certain choices of transformations, as previously explored in (Deco and Brauer, 1995; Dinh et al., 2014; Rezende and Mohamed, 2015; Kingma et al., 2016).”
- Neural Ordinary Differential Equations (2018, positive): “…theorem to compute exact changes in probability if samples are transformed through a bijective function : ⌅⌅⌅⌅ An example is the planar normalizing fl ow (Rezende and Mohamed, 2015): ⌅⌅ ⌅⌅ Generally, the main bottleneck to using the change of variables formula is computing of the determinant of…”
- A-NICE-MC: Adversarial Training for MCMC (2017, background): “A flow model [12, 26, 27] defines a generative model for x ∈ n through a bijection f : h → x , where h ∈ n have the same number of dimensions as x with a fixed prior p H ( h ) (typically a factored Gaussian).”
- Learning to Generate Samples from Noise through Infusion Training (2017, methodology): “…et al., 2015) • Continuous transformation of a distribution by invertible functions (Dinh et al. (2014), also used for variational inference in Rezende & Mohamed (2015)) Several of these approaches are based on maximizing an explicit or implicit model log-likelihood or a lower bound of its…”
- Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “Indeed, (Rezende and Mohamed, 2015) find that this type of transformation is generally less powerful than the normalizing flow presented above.”
- Density estimation using Real NVP (2016, background): “Recent work in this line of research has taken advantage of recent advances in recurrent networks [51], in particular long-short term memory [26], and residual networks [25, 24] in order to learn state-of-the-art generative image models [61, 46] and language models [32].”