Neural Spline Flows

By Conor Durkan, Artur Bekasov, Iain Murray, George Papamakarios. 2019

1,037 citations recorded by the source corpus.

A normalizing flow models a complex probability density as an invertible transformation of a simple base density. Flows based on either coupling or autoregressive transforms both offer exact density evaluation and sampling, but rely on the parameterization of an easily invertible elementwise transformation, whose choice determines the flexibility of these models. Building upon recent work, we propose a fully-differentiable module based on monotonic rational-quadratic splines, which enhances the flexibility of both coupling and autoregressive transforms while retaining analytic invertibility. We demonstrate that neural spline flows improve density estimation, variational inference, and generative modeling of images.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “, 2018), which are a subclass of autoregressive flows (Germain et al., 2015; Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and (ii) residual flows (Chen et al.”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “Recently, flow-based models (henceforth simply called flow models) have gained popularity as a type of deep generative model [7, 8, 32, 17, 3, 37, 68, 10] and for use in variational inference [34, 59, 33].”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “, 2018), which are a subclass of autoregressive flows (Germain et al., 2015; Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and (2) residual flows (Chen et al.”
Equivariant Flows: sampling configurations for multi-body systems with symmetric energies (2019, background): “a multivariate Normal distribution, via an invertible function fθ [4, 5, 14, 13, 10, 2, 7, 6].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “In image semantic segmentation, Huang et al. [207] proposed to generate fake images of the previous classes using a Scale-Aware Aggregation module and combine them with the images from the new task to facilitate the knowledge distillation process.”
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, methodology): “We also apply neural spline ﬂows (Durkan et al., 2019), which take the form of invertible nonlinear transformations by using monotonic rational-quadratic splines, to coupling layers.”
TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese (2020, methodology): “In future work we intend to investigate the training of flow-based [49, 50, 51] TTS models, such as Flowtron [20], GlowTTS [19] and Flow-TTS [52], in the TTS-Portuguese Corpus dataset.”
Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search (2020, background): “Flow-based generative models have received a lot of attention due to their advantages [7, 4, 21].”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “Alternating invertible linear transformations with autoregressive/coupling layers is often used in practice (see e.g. Kingma and Dhariwal, 2018; Durkan et al., 2019b).”
VFlow: More Expressive Generative Flows with Variational Data Augmentation (2020, background): “, 2019), flexible coupling functions (Ho et al., 2019; Durkan et al., 2019; Müller et al., 2019), and masked convolutional layers (Hoogeboom et al.”
OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport (2020, methodology): “Neural Spline Flows (Durkan et al. 2019) use splines instead of the coupling layers used in GLOW and RealNVP.”
IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression (2020, background): “This appears to be in contrast with many continuous ﬂows that are non-volume preserving such as afﬁne transformations [11], neural spline ﬂows [14] and Flow++ [17].”
A RAD approach to deep mixture models (2019, background): “, 2013) or modifications of spline functions (Müller et al., 2019; Durkan et al., 2019; Dolatabadi et al., 2020) that deliberately create discontinuities in the derivative.”
Bernstein flows for flexible posteriors in variational Bayes (2022, neutral): “However, there were also approaches proposed which use a single flexible transformation, such as sum-ofsquares polynomials [14], or splines [15].”
γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, methodology): “For all normalizing ﬂow experiments, we use a six-layer neural spline ﬂow (Durkan et al., 2019) with 16 knots deﬁned in the interval r´ 10 , 10 s .”
Liquid Time-constant Networks (2020, background): “2019) and to characterize them better (Dupont, Doucet, and Teh 2019; Durkan et al. 2019; Jia and Benson 2019; Hanshu et al. 2020; Holl, Koltun, and Thuerey 2020; Quaglino et al. 2020).”
Continuous diffusion for categorical data (2022, methodology): “Inspired by Durkan et al. (2019); Müller et al. (2019) we parameterise ˜ 𝐹 ( 𝑡 ) as a monotonic piecewise linear function, which is very straightforward to nor-malise and invert.”
SRFlow: Learning the Super-Resolution Space with Normalizing Flow (2020, background): “This challenge was first addressed in [10,11] and has recently spurred significant interest [5,14,21].”