Neural Ordinary Differential Equations
By T. Chen, Yulia Rubanova, Jesse Bettencourt, David Kristjanson Duvenaud. 2018
7,349 citations recorded by the source corpus.
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Ruslan Salakhutdinov
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrea Vedaldi
- Bernt Schiele
In Statistical and Nonlinear Physics
- Stefano Ermon
- Arash Vahdat
- Yang Song
- G. Em Karniadakis
- Jonathan Ho
- Karsten Kreis
- Jiaming Song
- Jun Zhu
In Model Architectures
- Yoshua Bengio
- Vladlen Koltun
- Aaron C. Courville
- David Kristjanson Duvenaud
- J. Kolter
- Christopher R'e
- Shaojie Bai
- Tri Dao
Citations in the corpus
- Denoising Diffusion Probabilistic Models (2020, background): “While diffusion models might resemble flows [9, 43, 10, 30, 5, 14, 21] and VAEs [31, 44, 34], diffusion models are designed so that q has no parameters and the top-level latent x T has nearly zero mutual information with the data x 0 .”
- RoFormer: Enhanced Transformer with Rotary Position Embedding (2021, methodology): “We look at several datasets from GLUE, i.e. MRPC Dolan and Brockett [2005], SST-2 Socher et al. [2013], QNLI Rajpurkar et al. [2016], STS-B Al-Natsheh [2017], QQP Chen et al. [2018b] and MNLI Williams et al. [2018].”
- Learning Deep Transformer Models for Machine Translation (2019, background): “In principle, residual networks can be seen as instances of the ordinary differential equation (ODE), behaving like the forward Euler discretization with an initial value (Chang et al., 2018; Chen et al., 2018b).”
- Stochastic Normalizing Flows (2020, background): “, 2019), and (ii) residual flows (Chen et al., 2018; Zhang et al., 2018; Grathwohl et al., 2018; Behrmann et al., 2018; Chen et al., 2019) – see (Papamakarios et al.”
- FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, methodology): “Recently, Chen et al. (2018) introduced a continuous-time analog of normalizing flows, defining the mapping from latent variables to data using ordinary differential equations (ODEs).”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “The red lines describe several smooth ODE trajectories from α = 0 to α = 1 corresponding to DDIM.”
- Score-based Generative Modeling in Latent Space (2021, methodology): “Sampling: To generate samples from LSGM at test time, we use a black-box ODE solver [73] to sample from the prior.”
- Score-Based Generative Modeling through Stochastic Differential Equations (2020, positive): “When the score function is approximated by the time-dependent score-based model, which is typically a neural network, this is an example of a neural ODE (Chen et al., 2018).”
- Denoising Diffusion Implicit Models (2020, background): “The sampling procedure of DDIM is also reminiscent of neural networks with continuous depth (Chen et al., 2018; Grathwohl et al., 2018), since the samples it produces from the same latent variable have similar high-level visual features, regardless of the specific sample trajectory.”
- Why Should We Add Early Exits to Neural Networks? (2020, background): “While a number of innovative components have been proposed recently (such as attention layers [ 71 ] , neural ODEs [ 17 ] , and graph modules [ 63 ] ), the vast majority of deep networks is designed as a sequential stack of (differentiable) layers, trained by propagating the gradient from the final…”
- Time-series forecasting with deep learning: a survey (2020, background): “While some preliminary research on continuoustime models has been done via neural ordinary differential equations [82], additional work needs to be done to extend this work for datasets with complex inputs (e.”
- Wireless Network Intelligence at the Edge (2018, background): “2) ODE-Based Training: Traditional ML cannot dynamically change the model size during the training process, so it is difficult to flexibly adjust the energy consumption.”
- The Incomplete Rosetta Stone problem: Identifiability results for Multi-view Nonlinear ICA (2019, background): “, the use of normalizing flows [33, 8] or deep invertible networks [24].”
- Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, methodology): “We compare the OTD implementation presented in (Chen et al., 2018; Grathwohl et al., 2018) using the dopri5-option (atol = 10−10,rtol = 10−5) to the DTO implementation given by Gholami et al.”
- Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, methodology): “Given an initial condition z0 and Fθ(z, t) = J∇zĤθ, we employ a twice-differentiable model architecture and a differentiable ODE solver (Chen et al., 2018) to compute predicted states {ẑ1, .”
- Equivariant Flows: sampling configurations for multi-body systems with symmetric energies (2019, methodology): “The solution of this initial value problem is x := y(t1) = y(t0) + ∫ t1 t0 dtgθ(y(t), t), which (under mild constraints) gives a bijection z → x As shown in [3], the change of density can be computed at any integration time, using the continuous change of variable rule, given by the differential equ”
- Equivariant Hamiltonian Flows (2019, background): “Normal-izing flows [26, 11, 5, 18, 4, 24, 16] provide a simple mechanism to build expressive density estimators in problems without much domain knowledge.”
- Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “The torchdiffeq (Chen, 2018) package is used.”
- Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “2 , and otherwise a segment of r % sequence length is masked, where r ∼ U [70 , 100] / U [10 , 100] . p uncond is set to 0 .”