Masked Autoregressive Flow for Density Estimation

By George Papamakarios, Iain Murray, Theo Pavlakou. 2017

1,636 citations recorded by the source corpus.

Autoregressive models are among the best performing neural density estimators. We describe an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses internally when generating data. By constructing a stack of autoregressive models, each modelling the random numbers of the next model in the stack, we obtain a type of normalizing flow suitable for density estimation, which we call Masked Autoregressive Flow. This type of flow is closely related to Inverse Autoregressive Flow and is a generalization of Real NVP. Masked Autoregressive Flow achieves state-of-the-art performance in a range of general-purpose density estimation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

Jukebox: A Generative Model for Music (2020, background): “Autoregressive flows (Papamakarios et al., 2017; Kingma et al., 2016) combine the two ideas to achieve faster density estimation or data generation.”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, methodology): “Autoregressive models (or flows (Papamakarios et al., 2017)) model the data as p(xt|x<t) and IAFs as p(xt|z<t).”
Stochastic Normalizing Flows (2020, background): “…layers (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; M ¨ uller et al., 2018), which are a subclass of autoregressive ﬂows (Germain et al., 2015; Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and (ii) residual ﬂows (Chen et al., 2018; Zhang et al.,…”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, neutral): “As a point of comparison, we include a similar result from a 5 layer Masked Autoregressive Flow distribution [58].”
FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, methodology): “These models require O ( D ) sequential computations to sample from while the best performing method, MAF-DDSF (Huang et al., 2018), cannot be sampled from analytically.”
Glow: Generative Flow with Invertible 1x1 Convolutions (2018, background): “In (Papamakarios et al., 2017) (MAF), the authors propose a generative flow based on IAF (Kingma et al.”
Autoregressive Quantile Networks for Generative Modeling (2018, methodology): “Gaussian) base distribution, the ordering of the dimensions can be crucial (Papamakarios et al., 2017).”
GAN Inversion: A Survey (2021, methodology): “Classic autoregressive flow models include MAF [173] and IAF [174].”
The frontier of simulation-based inference (2019, background): “Proceedings of the National Academy of Sciences of the United States of America 100(26):15324–15328.”
High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes (2019, methodology): “Deep learning-based sequence models have been successfully applied to audio signals [33], language modeling [13, 30], and general density estimation of univariate sequences [22, 21].”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “…layers (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; Müller et al., 2018), which are a subclass of autoregressive flows (Germain et al., 2015; Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and 2) residual flows (Chen et al., 2018; Zhang et al.,…”
Equivariant Flows: sampling configurations for multi-body systems with symmetric energies (2019, background): “a multivariate Normal distribution, via an invertible function fθ [4, 5, 14, 13, 10, 2, 7, 6].”
Equivariant Hamiltonian Flows (2019, background): “Normalizing flows [26, 11, 5, 18, 4, 24, 16] provide a simple mechanism to build expressive density estimators in problems without much domain knowledge.”
A Survey on Neural Speech Synthesis (2021, background): “AR AF [255] zt = xt · σt(x<t; θ) + μt(x<t; θ) xt = zt−ut(x<t;θ) σt(x<t;θ) IAF [165] zt = xt−μt(z<t;θ) σt(z<t;θ) xt = zt · σt(z<t; θ) + μt(z<t; θ)”
Invertible Residual Networks (2018, neutral): “67 MAF (Papamakarios et al., 2017) 1.”
Neural Spline Flows (2019, background): “[41], who make their data available online [40].”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “, 2016), masked autoregressive flow (MAF) (Papamakarios et al., 2017), and Glow (Kingma and Dhariwal, 2018).”
FloWaveNet : A Generative Flow for Raw Audio (2018, methodology): “The neural density estimator family has the advantage of using a much more flexible transformation, such as masked autoregressive flow (Papamakarios et al., 2017) and transformation autoregressive network (Oliva et al.”
Neural Autoregressive Flows (2018, methodology): “, 2016) or better density estimation for generative models (Papamakarios et al., 2017).”
WaveFlow: A Compact Flow-based Model for Raw Audio (2019, methodology): “In contrast, IAF uses an autoregressive transformation for inverse mapping z = f−1(x): zt = xt − µt(z<t;ϑ) σt(z<t;ϑ) , (3) making density evaluation very slow for likelihood-based training, but one can sample x = f(z) in parallel through xt = zt · σt(z<t;ϑ) + µt(z<t;ϑ).”