Sylvester Normalizing Flows for Variational Inference

By Rianne van den Berg, Leonard Hasenclever, J. Tomczak, Max Welling. 2018

260 citations recorded by the source corpus.

Variational inference relies on flexible approximate posterior distributions. Normalizing flows provide a general recipe to construct flexible variational posteriors. We introduce Sylvester normalizing flows, which can be seen as a generalization of planar flows. Sylvester normalizing flows remove the well-known single-unit bottleneck from planar flows, making a single transformation much more flexible. We compare the performance of Sylvester normalizing flows against planar flows and inverse autoregressive flows and demonstrate that they compare favorably on several datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “Moreover, having access to tractable density is useful in variational inference (Rezende & Mohamed, 2015; Tomczak & Welling, 2016; Louizos & Welling, 2017; Berg et al., 2018) or approximate sampling from distributions given by an energy function (Oord et al., 2017).”
FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, methodology): “By restricting the functional form of f , various determinant identities can be exploited (Rezende & Mohamed, 2015; Berg et al., 2018).”
Neural Ordinary Differential Equations (2018, neutral): “Recent work explores the tradeoff between the expressiveness of normalizing ﬂ ow layers and computational cost (Kingma et al., 2016; Tomczak and Welling, 2016; Berg et al., 2018).”
Diagnosing and Enhancing VAE Models (2019, background): “In particular, multiple elegant approaches involve replacing the Gaussian encoder network with a richer class of distributions instantiated through normalizing ﬂows or related (Burda et al., 2015; Kingma et al., 2016; Rezende and Mohamed, 2015; van den Berg et al., 2018).”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “While flows can be used to build generative models by maximizing the likelihood on a data sample, having access to tractable density is especially useful in variational inference (Rezende & Mohamed, 2015; Tomczak & Welling, 2016; Louizos & Welling, 2017; Berg et al., 2018) or approximate sampling fr”
Sorting out Lipschitz function approximation (2018, background): “[51] to enforce orthogonal weights for flexible variational inference with normalizing flows [39].”
Neural Spline Flows (2019, background): “In recent years, normalizing flows have received widespread attention in the machine-learning literature, seeing successful use in density estimation [9, 41], variational inference [29, 35, 44, 55], image, audio and video generation [25, 27, 31, 43], likelihood-free inference [42], and learning maxi”
Neural Autoregressive Flows (2018, background): “There Finally, there are also several works which develop new techniques for constructing NFs that are orthogonal to ours (Tomczak & Welling, 2017; 2016; Gemici et al., 2016; Du-venaud et al., 2016; Berg et al., 2018).”
WaveFlow: A Compact Flow-based Model for Raw Audio (2019, background): “Flow-based models can either represent the approximate posteriors for variational inference (Rezende & Mohamed, 2015; Kingma et al., 2016; Berg et al., 2018), or can be trained directly on data using the change of variables formula (Dinh et al.”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, methodology): “For the results in this section, we use the architecture from Berg et al. [30] to encode and decode z mk .”
Taming VAEs (2018, background): “As a result, several mechanisms have been proposed to increase the expressiveness of the variational posterior density [21, 22, 23, 24, 25, 3, 26, 5] but it still remains a challenge to train complex encoders due to a combination of optimization and generalization issues [27, 28].”
Analyzing Inverse Problems with Invertible Neural Networks (2018, methodology): “Besides variants of the original architectures [43, 2, 44], the most common realizations use auto-regressive flows, where the density is decomposed according to the Bayesian chain rule [23, 17, 12, 31, 29, 25, 38, 45].”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, methodology): “…as (1) an inference machine in the context of variational inference for continuous latent variable models (Kingma et al., 2016; Tomczak & Welling, 2016; Berg et al., 2018), and (2) a trainable component of the latent variable model (Chen et al., 2017; Agrawal & Dukkipati, 2016; Huang et al., 2017).”
Backpropagation for Implicit Spectral Densities (2018, methodology): “[2016c] apply this idea to image generation, with follow-up work (Dinh et al. [2016], van den Oord et al. [2016b]) that exploits parallelism using masked convolutions. van den Oord et al. [2016a] do the same for audio generation, and van den Oord et al. [2017] introduce strategies to improve efficie”
MintNet: Building Invertible Neural Networks with Masked Convolutions (2019, background): “For example, in Sylvester flow [2], the invertible transformation has the form f(x) , x + Ah(Bx + b), where h(·) is a nonlinear activation function, A ∈ RD×M , B ∈ RM×D, b ∈ R and M ≤ D.”
Hamiltonian Variational Auto-Encoder (2018, background): “NFs have demonstrated success in various domains [2, 16], but the flows do not explicitly use information about the target posterior.”
Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, unknown): “These matrices allow for efﬁcient computation of inverses and their Jacobian determinant of one, making them particularly suitable for applications in normalizing ﬂows [64, 10].”
Unconstrained Monotonic Neural Networks (2019, methodology): “38 Sylvester - Berg et al. [2018] 83.”