A Family of Nonparametric Density Estimation Algorithms

By Esteban G. Tabak, Cristina Vilma Turner. 2013

553 citations recorded by the source corpus.

A new methodology for density estimation is proposed. The methodology, which builds on the one developed by Tabak and Vanden‐Eijnden, normalizes the data points through the composition of simple maps. The parameters of each map are determined through the maximization of a local quadratic approximation to the log‐likelihood. Various candidates for the elementary maps of each step are proposed; criteria for choosing one includes robustness, computational simplicity, and good behavior in high‐dimensional settings. A good choice is that of localized radial expansions, which depend on a single parameter: all the complexity of arbitrary, possibly convoluted probability densities can be built through the composition of such simple maps. © 2012 Wiley Periodicals, Inc.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Stochastic Normalizing Flows (2020, methodology): “Normalizing ﬂows (NF) are popular generative learning methods (Tabak et al., 2010; Tabak & Turner, 2013; Dinh et al., 2014; Rezende & Mohamed, 2015).”
Variational Inference with Normalizing Flows (2015, background): “One path towards this ideal is based on the principle of normalizing ﬂows (Tabak & Turner, 2013; Tabak & Vanden-Eijnden, 2010).”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “Normalizing flows Normalizing flows (NFs) are diffeomorphisms fθ : Rn → Rn which transform samples z ∼ ρ from a simple prior density ρ into samples x = fθ(z) (Tabak et al., 2010; Tabak & Turner, 2013; Rezende & Mohamed, 2015; Papamakarios et al., 2019).”
Do Deep Generative Models Know What They Don't Know? (2018, background): “…which take many forms such as variational auto-encoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), generative adversarial networks (GANs) (Goodfel-∗ low et al., 2014), auto-regressive models (van den Oord et al., 2016b;a), and invertible latent variable models (Tabak & Turner, 2013).”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “Early work (Chen and Gopinath, 2000; Tabak and Turner, 2013) considered only synthetic low-dimensional cases, showing that normalizing flows could indeed represent skewed, multi-modal densities as well as kernel density estimators could.”
Sylvester Normalizing Flows for Variational Inference (2018, neutral): “The resulting transformed density after one such transformation f is as follows (Tabak and Turner, 2013; Tabak and Vanden-Eijnden, 2010): p1(z ′) = p0(z) ∣∣∣∣det(∂f(z)∂z )∣∣∣∣−1 , (4) where z′ = f(z), z, z′ ∈ RD and f : RD 7→ RD is an invertible function.”
Improving Variational Auto-Encoders using Householder Flow (2016, background): “A (finite) normalizing flow, first formulated by [20, 21] and further developed by [14], is a powerful framework for building flexible posterior distribution by starting with an initial random variable”
Energy-based Out-of-distribution Detection (2020, positive): “Generative models [7, 17, 34, 38, 40] can be alternative approaches for detecting OOD examples, as they directly estimate the in-distribution density and can declare a test sample to be out-of-distribution if it lies in the low-density regions.”
Analyzing Inverse Problems with Invertible Neural Networks (2018, background): “Today, this approach is often formulated as a normalizing flow (Tabak et al., 2010; Tabak and Turner, 2013), which gradually transforms a normal density into the desired data density and relies on bijectivity to ensure the mapping’s validity.”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, positive): “The term Normalizing Flow was originally coined by Tabak et al. (2010); Tabak & Turner (2013) where it was used for density estimation.”
Discovering Interpretable Representations for Both Deep Generative and Discriminative Models (2018, methodology): “In order to transform the density of z into z∗, we use normalizing flows (Tabak & Turner, 2013; Tabak & VandenEijnden, 2010; Rezende & Mohamed, 2015; Kingma et al., 2016), which are powerful and invertible transformations used to construct flexible posterior distributions through a series of…”
OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport (2020, background): “Related Works Finite Flows Normalizing flows (Tabak and Turner 2013; Rezende and Mohamed 2015; Papamakarios et al. 2019; Kobyzev, Prince, and Brubaker 2020) use a composition of discrete transformations, where specific architectures are chosen to allow for efficient inverse and Jacobian determinant ”
On Memorization in Probabilistic Deep Generative Models (2021, background): “In the last few years there have been incredible successes in generative modeling through the development of deep learning techniques such as variational autoencoders (VAEs) [1, 2], generative adversarial networks (GANs) [3], normalizing flows [4, 5], and diffusion networks [6, 7], among others.”
Discrete Flows: Invertible Generative Models of Discrete Data (2019, methodology): “There have been many recent advances in normalizing ﬂows, a technique for constructing high-dimensional continuous distributions from invertible transformations of simple distributions (Rezende and Mohamed, 2015; Tabak and Turner, 2013; Rippel and Adams, 2013).”
Detecting Out-of-Distribution Inputs to Deep Generative Models Using a Test for Typicality (2019, background): “Examples of deep generative models that meet these speciﬁcations include normalizing ﬂows [36] such as Glow [22], latent variable models such as variational autoencoders (VAEs) [23, 29], and auto-regressive models such as PixelCNN [38].”
Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality (2019, background): “Examples of DGMs that meet these specifications include normalizing flows [56] such as Glow [32], latent variable models such as variational autoencoders (VAEs) [33, 47], and auto-regressive models such as PixelCNN [58].”
A RAD approach to deep mixture models (2019, methodology): “…variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014), generative adversarial networks (Goodfellow et al., 2014), and ﬂow based models (Baird et al., 2005; Tabak & Turner, 2013; Dinh et al., 2015; 2017; Kingma & Dhari-wal, 2018; Chen et al., 2018; Grathwohl et al., 2019).”
Why Normalizing Flows Fail to Detect Out-of-Distribution Data (2020, background): “Normalizing flows Normalizing flows [42] are a flexible class of deep generative models that model a target distribution p⇤(x) as an invertible transformation f of a base distribution pZ(z) in the latent space.”
Integer Discrete Flows and Lossless Compression (2019, background): “A composition of invertible functions is generally referred to as a normalizing flow in deep learning literature, which can be viewed as a repeated application of the change of variables formula [5, 31, 30, 24].”