Residual Flows for Invertible Generative Modeling

By Ricky T. Q. Chen, Jens Behrmann, David Kristjanson Duvenaud, Jörn-Henrik Jacobsen. 2019

450 citations recorded by the source corpus.

Flow-based generative models parameterize probability distributions through an invertible transformation and can be trained by maximum likelihood. Invertible residual networks provide a flexible family of transformations where only Lipschitz conditions rather than strict architectural constraints are needed for enforcing invertibility. However, prior work trained invertible residual networks for density estimation by relying on biased log-density estimates whose bias increased with the network's expressiveness. We give a tractable unbiased estimate of the log density using a "Russian roulette" estimator, and reduce the memory required during training by using an alternative infinite series for the gradient. Furthermore, we improve invertible residual blocks by proposing the use of activation functions that avoid derivative saturation and generalizing the Lipschitz condition to induced mixed norms. The resulting approach, called Residual Flows, achieves state-of-the-art performance on density estimation amongst flow-based models, and outperforms networks that use coupling blocks at joint generative and discriminative modeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “…Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and (ii) residual ﬂows (Chen et al., 2018; Zhang et al., 2018; Grathwohl et al., 2018; Behrmann et al., 2018; Chen et al., 2019) – see (Papamakarios et al., 2019) for an extensive review of ﬂow architectures.”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, methodology): “Recent work (Behrmann et al., 2018; Chen et al., 2019) attempts to improve the discriminative performance of generative models by leveraging invertible architectures, but these methods still underperform their purely discriminative counterparts jointly trained as generative models.”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, neutral): “…et al., 2016) 3.49 - iResNet (Behrmann et al., 2019) 3.45 - Glow (Kingma & Dhariwal, 2018) 3.35 - MintNet (Song et al., 2019b) 3.32 - Residual Flow (Chen et al., 2019) 3.28 46.37 FFJORD (Grathwohl et al., 2018) 3.40 - Flow++ (Ho et al., 2019) 3.29 - DDPM ( L ) (Ho et al., 2020) ď 3.70 * 13.51 DDPM…”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “…al., 2018), which are a subclass of autoregressive flows (Germain et al., 2015; Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and 2) residual flows (Chen et al., 2018; Zhang et al., 2018; Grathwohl et al., 2018; Behrmann et al., 2018; Chen et al., 2019).”
VFlow: More Expressive Generative Flows with Variational Data Augmentation (2020, methodology): “Residual Flow (Chen et al., 2019).”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, methodology): “…in probability density and have been widely applied in many tasks, including (i) generative models (Dinh et al., 2017; Kingma & Dhari-wal, 2018; Chen et al., 2019), (ii) variational inference (Rezende & Mohamed, 2015; Kingma et al., 2016; Berg et al., 2018), (iii) density estimation…”
Normalizing Flows: An Introduction and Review of Current Methods (2020, unknown): “, 2019], Residual Flow [Chen et al., 2019], Flow++ [Ho et al.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “The authors originally proposed a biased approximation of the log determinant of the Jacobian as a power series where the Jacobian trace is approximated using Hutchkinson’s trace estimator (see Table 3), but an unbiased approximator known as a Russian roulette estimator has also been proposed [25].”
Moser Flow: Divergence-based Generative Modeling on Manifolds (2021, background): “First, we consider a collection of challenging toy 2D datasets explored in prior works (Chen et al., 2020; Huang et al., 2021).”
Semi-Supervised Learning with Normalizing Flows (2019, methodology): “Moreover, recent work (Dinh et al., 2016; Kingma & Dhariwal, 2018; Behrmann et al., 2018; Chen et al., 2019; Song et al., 2019) demonstrated that normalizing flows can produce high-fidelity samples for natural image datasets.”
Hamiltonian Generative Networks (2019, background): “…2015; Kingma et al., 2016; Papamakarios et al., 2017; Dinh et al., 2017; Huang et al., 2018; Kingma & Dhariwal, 2018; Hoogeboom et al., 2019; Chen et al., 2019; Grathwohl et al., 2018), none of the approaches manage to produce both sampling and density evaluation steps that are…”
Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows (2019, methodology): “Unlike models based on triangular maps (Jaini et al., 2019), ResFlows have the attractive feature that the structure of their Jacobians is unconstrained, which may explain their greater expressiveness.”
Hybrid Discriminative-Generative Training via Contrastive Learning (2020, methodology): “where α is weight between [0, 1].”