FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

By Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, David Kristjanson Duvenaud. 2018

1,080 citations recorded by the source corpus.

A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network. Likelihood-based training of these models requires restricting their architectures to allow cheap computation of Jacobian determinants. Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson's trace estimator to give a scalable unbiased estimate of the log-density. The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures. We demonstrate our approach on high-dimensional density estimation, image generation, and variational inference, achieving the state-of-the-art among exact likelihood methods with efficient sampling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, background): “While diffusion models might resemble ﬂows [9, 43, 10, 30, 5, 14, 21] and VAEs [31, 44, 34], diffusion models are designed so that q has no parameters and the top-level latent x T has nearly zero mutual information with the data x 0 .”
Stochastic Normalizing Flows (2020, background): “…Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and (ii) residual ﬂows (Chen et al., 2018; Zhang et al., 2018; Grathwohl et al., 2018; Behrmann et al., 2018; Chen et al., 2019) – see (Papamakarios et al., 2019) for an extensive review of ﬂow architectures.”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “Recently, ﬂow-based models (henceforth simply called ﬂow models ) have gained popularity as a type of deep generative model [7, 8, 32, 17, 3, 37, 68, 10] and for use in variational inference [34, 59, 33].”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “We also show that the learned short-run MCMC can be used as a generative model, such as a generator model [13, 24] or the flow model [8, 9, 25, 4, 14], with the Langevin dynamics serving as a noiseinjected residual network, with the initial image serving as the latent variables, and with the initial”
Score-based Generative Modeling in Latent Space (2021, methodology): “The stochasticity in the estimation comes from Hutchinson’s trick [106].”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, positive): “In many cases computing ∇ ¨ ˜ f θ p x , t q is expensive, so we follow Grathwohl et al. (2018) to estimate it with the Skilling-Hutchinson trace estimator (Skilling, 1989; Hutchinson, 1990).”
Denoising Diffusion Implicit Models (2020, neutral): “The sampling procedure of DDIM is also reminiscent of neural networks with continuous depth (Chen et al., 2018; Grathwohl et al., 2018), since the samples it produces from the same latent variable have similar high-level visual features, regardless of the speciﬁc sample trajectory.”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, methodology): “Finally, computing the divergence of v using off-the-shelf AD frameworks requires O(n) backpropagation passes, which would result in an infeasible overhead for highdimensional systems (Grathwohl et al., 2018).”
Equivariant Flows: sampling configurations for multi-body systems with symmetric energies (2019, background): “a multivariate Normal distribution, via an invertible function fθ [4, 5, 14, 13, 10, 2, 7, 6].”
Flow Matching for Generative Modeling (2022, background): “Although some works demonstrated the capability of CNF generative models for image synthesis (Grathwohl et al., 2018), scaling up to very high dimensional images is inherently difficult.”
Excessive Invariance Causes Adversarial Vulnerability (2018, background): “, 2018) and that iRevNets as well as Glow-type networks are related to ResNets by the choice of dimension splitting applied in their residual blocks (Grathwohl et al., 2018).”
Invertible Residual Networks (2018, methodology): “However, in this indirect approach the dynamics depend on the discretization, while the direct approach [7, 13] learns the ODE and adapts the discretization based on the ﬁxed ODE.”
Neural Spline Flows (2019, background): “The resulting continuous-time flow is known as FFJORD [15].”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “Continuous (Grathwohl et al., 2019) and residual (Behrmann et al., 2019; Chen et al., 2019) flows have been demonstrated to produce sharp, high-dimensional images as well.”
WaveFlow: A Compact Flow-based Model for Raw Audio (2019, background): “, 2018), or can be trained directly on data using the change of variables formula (Dinh et al., 2014; 2017; Kingma and Dhariwal, 2018; Grathwohl et al., 2018).”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, methodology): “e normalizing ows can be generalized into a continuous formulation (Chen et al. 2018; Grathwohl et al. 2018) using neural ODE (Chen et al.”
Residual Flows for Invertible Generative Modeling (2019, background): “, 2014, 2017) or solving an ordinary differential equation (Grathwohl et al., 2019).”