Unconstrained Monotonic Neural Networks

By Antoine Wehenkel, Gilles Louppe. 2019

182 citations recorded by the source corpus.

Monotonic neural networks have recently been proposed as a way to define invertible transformations. These transformations can be combined into powerful autoregressive flows that have been shown to be universal approximators of continuous probability distributions. Architectures that ensure monotonicity typically enforce constraints on weights and activation functions, which enables invertibility but leads to a cap on the expressiveness of the resulting transformations. In this work, we propose the Unconstrained Monotonic Neural Network (UMNN) architecture based on the insight that a function is monotonic as long as its derivative is strictly positive. In particular, this latter condition can be enforced with a free-form neural network whose only constraint is the positiveness of its output. We evaluate our new invertible building block within a new autoregressive flow (UMNN-MAF) and demonstrate its effectiveness on density estimation experiments. We also illustrate the ability of UMNNs to improve variational inference.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “For example, Wehenkel and Louppe (2019) define the transformer as: τ(zi;hi) = ∫ zi 0 g(z;αi) dz + βi where hi = {αi, βi} , (35) where g(·;αi) can be any positive-valued neural network parameterized by αi.”
OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport (2020, methodology): “Different integration schemes have various error when integrating the trace (Wehenkel and Louppe 2019; Onken and Ruthotto 2020).”
Normalizing Flows: An Introduction and Review of Current Methods (2020, methodology): “For experimental details, see the following papers: RealNVP [20] and MAF [74], Glow [57] and FFJORD [34], NAF [45], UMNN [102], SOS [49], Quadratic Spline ﬂow and RQ-NSF [25], Cubic Spline Flow [24].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “bility to achieve this is to define h as an integral over a positive but otherwise unconstrained function g [242],”