Neural Autoregressive Flows

By Chin-Wei Huang, David Krueger, Alexandre Lacoste, Aaron C. Courville. 2018

495 citations recorded by the source corpus.

Normalizing flows and autoregressive models have been successfully combined to produce state-of-the-art results in density estimation, via Masked Autoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-based speech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows (IAF). We unify and generalize these approaches, replacing the (conditionally) affine univariate transformations of MAF/IAF with a more general class of invertible univariate transformations expressed as monotonic neural networks. We demonstrate that the proposed neural autoregressive flows (NAF) are universal approximators for continuous probability distributions, and their greater expressivity allows them to better capture multimodal target distributions. Experimentally, NAF yields state-of-the-art performance on a suite of density estimation tasks and outperforms IAF in variational autoencoders trained on binarized MNIST.

Subfields: Generative Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Model Architectures

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “…2016; Kingma & Dhariwal, 2018; M ¨ uller et al., 2018), which are a subclass of autoregressive ﬂows (Germain et al., 2015; Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and (ii) residual ﬂows (Chen et al., 2018; Zhang et al., 2018; Grathwohl et al.,…”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, neutral): “Huang et al. (2018) show how to design”
FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, methodology): “These models require O ( D ) sequential computations to sample from while the best performing method, MAF-DDSF (Huang et al., 2018), cannot be sampled from analytically.”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, background): “…ON INVERSE AUTOREGRESSIVE FLOW Inverse autoregressive flows (IAF, Kingma et al. (2016)) and are similar to very deep VAEs in that they are universal approximators of posterior distributions in VAEs, even with just a single layer and sufficiently expressive univariate density (Huang et al., 2018).”
Differentiable Causal Discovery from Interventional Data (2020, methodology): “In our implementation, we use deep sigmoidal flows (DSF), a specific instantiation of normalizing flows which is a universal density approximator [18].”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “…2016; Kingma & Dhariwal, 2018; Müller et al., 2018), which are a subclass of autoregressive flows (Germain et al., 2015; Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and 2) residual flows (Chen et al., 2018; Zhang et al., 2018; Grathwohl et al., 2018;…”
A Survey on Neural Speech Synthesis (2021, background): “IAF can be regarded as a dual formulation of autoregressive flow (AF) [255, 121].”
Neural Spline Flows (2019, background): “Subsequent work which enhances their flexibility has resulted in models which do not have an analytic inverse, and require numerical optimization to invert [21].”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “The class of autoregressive flows are a direct implementation of this construction, specifying fφ to have the following form (as described by e.g. Huang et al., 2018; Jaini et al., 2019): z′i = τ(zi;hi) where hi = ci(z<i), (28) where τ is termed the transformer and ci the i-th conditioner.”
WaveFlow: A Compact Flow-based Model for Raw Audio (2019, background): “One group of models are based on autoregressive transformation, including autoregressive flow (AF) and inverse autoregressive flow (IAF) as the “dual” of each other (Kingma et al., 2016; Papamakarios et al., 2017; Huang et al., 2018).”
Resampled Priors for Variational Autoencoders (2018, methodology): “[4] used autoregressive flows to define priors which are both fairly fast to evaluate and sample from.”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, methodology): “For example, one can generalize the Ornstein-Uhlenbeck process to have non-linear (in time) f and g , similar to the variance-presering SDE, by parameterizing the integral of f using a monotone network (Sill, 1998; Kay & Ungar, 2000; Daniels & Velikova, 2010; Huang et al., 2018).”
Neural Importance Sampling (2018, background): “A number of recent approaches investigate the usage of normalizing flows for auto-regressive density estimation (Huang et al. 2018; Kingma et al. 2016; Papamakarios et al. 2017).”
Variational Autoencoder with Implicit Optimal Priors (2018, methodology): “The normalizing ﬂow (Rezende and Mohamed 2015; Kingma et al. 2016; Huang et al. 2018) is one of the main approaches, which applies a sequence of invertible transformations to the latent variable vector until a desired level of ﬂexibility is attained.”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, background): “, 2018), (iii) density estimation (Papamakarios et al., 2017; Huang et al., 2018), (iv) reinforcement learning (Mazoure et al.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “namely Neural Autoregressive Flows (NAF) [95] and Block NAF [38].”
Diffusion Normalizing Flow (2021, methodology): “Compared with autoregressive models, it excels NAF [18] on all but GAS.”