Normalizing Flows for Probabilistic Modeling and Inference

By George Papamakarios, Eric T. Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, Balaji Lakshminarayanan. 2019

2,415 citations recorded by the source corpus.

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “2 with any invertible differentiable transformation, such as NICE (Dinh et al., 2014), RealNVP (Dinh et al., 2016), invertible ResNets (Behrmann et al., 2018) or neural ODEs (Chen et al., 2018) – see (Papamakarios et al., 2019) for an overview of ﬂow architectures.”
Generative Adversarial Networks (2021, neutral): “We have discussed the key design choices and variations in the GAN architecture, as well as the different LFs utilized to train GAN models.”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, methodology): “We omit full proof, and refer readers to Huang et al. (2017); Papamakarios et al. (2019), where universality is established for autoregressive flows.”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “Gradient flows (not necessarily G-equivariant) can map any ρ onto any ρ′ over Rn as long as both densities do not vanish (Benamou & Brenier, 2000; McCann, 2001) and have been discussed in the context of density estimation (Zhang et al., 2018; Papamakarios et al., 2019).”
Posterior Meta-Replay for Continual Learning (2021, methodology): “Normalizing ﬂows [57] can also be utilized as WG architectures to gain analytic access to q θ ( t ) ( W ) , albeit at the cost of architectural constraints such as invertibility.”
VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, methodology): “Hence, some research uses normalizing flow [21] in AM, such as FlowTTS [22] and GlowTTS [23].”
A Survey on Neural Speech Synthesis (2021, methodology): “The flow-based models used in neural TTS can be divided into two categories according to the two different techniques [256]: 1) autoregressive transforms [165] (e.”
End-to-End Adversarial Text-to-Speech (2020, background): “More recently, flow-based models [42] are being explored as a feed-forward alternative that enables fast inference (without sequential dependencies).”
Learning Gradient Fields for Shape Generation (2020, neutral): “…loss λ ( σ i ) ; Point cloud encoder f φ ; A neural network s θ ; Total number of training iterations T ; Point cloud X t 1: for t ← 1 to T do for σ ∈ { σ i } ki =1 do 4: x X for i ∈ t do 5: ˜ x i ← x i + N (0 , σ 2 I ) for end 7: ( σ, 8: end for 9: } φ, θ ← Adam ( , φ, θ ) for t ← 0 to T − 1 do”
Structured Denoising Diffusion Models in Discrete State-Spaces (2021, methodology): “There has been a great deal of progress with the development of methods like GANs [14, 3], VAEs [22, 32], large autoregressive neural network models [43, 42, 44], normalizing flows [31, 11, 21, 30], and others, each with their own tradeoffs in terms of sample quality, sampling speed, log-likelihoods”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, background): “For a comprehensive review of this topic, see Kobyzev et al. (2019) and Papamakarios et al. (2019).”
IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression (2020, methodology): “[38] for a recent comprehensive review of the field.”
Deep Structural Causal Models for Tractable Counterfactual Inference (2020, methodology): “For an observed variable x, diffeomorphic transformation f , and base variable ∼ P ( ) such that x = f( ), the output density p(x) can be computed as p(x) = p( )|det∇f( )|−1, evaluated at = f−1(x) [21, 22].”
Behavior-Driven Synthesis of Human Dynamics (2021, methodology): “To alleviate this issue, we train a normalizing flow model [44, 33] after our variational behavior model is optimized.”
OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport (2020, methodology): “They perform well in density estimation (Chen et al. 2018a; Grathwohl et al. 2019; Papamakarios et al. 2019) and inference (Ingraham et al. 2019; Papamakarios et al. 2019), especially in physics and computational chemistry (Noé et al. 2019; Brehmer et al. 2020).”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “In addition, we discuss the connections of diffusion models to other deep generative models (Section 6), including variational autoencoders (VAEs) [123, 195], generative adversarial networks (GANs) [73], normalizing flows [50, 52, 175, 197], autoregressive models [237], and energy-based models (EBMs”