Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit

By Belinda Tzen, Maxim Raginsky. 2019

268 citations recorded by the source corpus.

In deep latent Gaussian models, the latent variable is generated by a time-inhomogeneous Markov chain, where at each time step we pass the current state through a parametric nonlinear map, such as a feedforward neural net, and add a small independent Gaussian perturbation. This work considers the diffusion limit of such models, where the number of layers tends to infinity, while the step size and the noise variance tend to zero. The limiting latent object is an Ito diffusion process that solves a stochastic differential equation (SDE) whose drift and diffusion coefficient are implemented by neural nets. We develop a variational inference framework for these \textit{neural SDEs} via stochastic automatic differentiation in Wiener space, where the variational approximations to the posterior are obtained by Girsanov (mean-shift) transformation of the standard Wiener process and the computation of gradients is based on the theory of stochastic flows. This permits the use of black-box SDE solvers and automatic differentiation for end-to-end inference. Experimental results with synthetic data are provided.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “Related model designs include the work of [70, 32, 33].”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, methodology): “…divergence between two SDEs with shared diffusion coefﬁcients and starting points exists under assumptions (vii) (viii) (ix) (x) (xi) (see, e.g ., [52, 30]), and can be computed via the Girsanov theorem [34]: where (i) is due to Girsanov Theorem II [34, Theorem 8.6.6], and (ii) is due to the…”
Video Diffusion Models (2022, background): “A diffusion model [46, 47, 22] speciﬁed in continuous time [53, 48, 10, 28] is a generative model with latents z = { z t | t ∈ [0 , 1] } obeying a forward process q ( z | x ) starting at data x ∼ p ( x ) .”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, methodology): “Before we make the connection to score matching, we formally address the common belief that “diffusion models can be viewed as the continuous limit of hierarchical VAEs” (Tzen & Raginsky, 2019), and show that the CT-ELBO consistently extends their discrete-time counterpart.”
Normalizing Flows: An Introduction and Review of Current Methods (2020, methodology): “Several papers have worked explicitly with the SDE [14], [62], [63], [76], [96].”
GRAND: Graph Neural Diffusion (2021, background): “Many follow-up works explored augmentation (Dupont et al., 2019) and regularization (Finlay et al., 2020) and provided extensions into new domains such as stochastic (Li et al., 2020a; Tzen & Raginsky, 2019) differential equations.”
Scalable Gradients for Stochastic Differential Equations (2020, methodology): “This approach should not be confused with the formulation of variational inference for non-discretized SDEs presented in previous works [24, 54, 79] and our work, as it is unclear whether the limit of their discretization corresponds to that obtained by operating with continuous-time SDEs using Girs”
Stochastic Normalizing Flows (2020, methodology): “Stochastic extensions of neural ODEs have been considered in (Tzen & Raginsky, 2019; Liu et al., 2019; Jia & Benson, 2019; Peluchetti & Favaro, 2019) as limits of deep latent Gaussian models, where they have been suggested to show increased robustness to noisy / adversarial data.”
Neural Controlled Differential Equations for Irregular Time Series (2020, methodology): “Adjoint backpropagationneeds some work to apply to SDEs, and so [42, 43, 44] all propose methods for training Neural SDEs.”
Neural Jump Stochastic Differential Equations (2019, methodology): “proposed an RNN-based variational method for learning stochastic dynamics [39], which has later been generalized for infinitesimal step size [40, 41].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “By modelling the Langevin sampling procedure as a stochastic differential equation (SDE), pre-existing SDE solvers can be used, reducing the number of steps required [182], [200].”
Imagen Video: High Definition Video Generation with Diffusion Models (2022, methodology): “Imagen Video is built from diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020) speciﬁed in continuous time (Tzen & Raginsky, 2019; Song et al., 2021; Kingma et al., 2021).”
MAGVIT: Masked Generative Video Transformer (2022, background): “For example, the state-of-the-art video diffusion model [33] extends the image denoising diffusion model [3,32,52,54,60] by incorporating 3D U-Net [14] architectures and joint training on both images and videos.”