Neural Ordinary Differential Equations

By T. Chen, Yulia Rubanova, Jesse Bettencourt, David Kristjanson Duvenaud. 2018

7,349 citations recorded by the source corpus.

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Sergey Levine
  3. Mohit Bansal
  4. Ruslan Salakhutdinov
  5. Quoc V. Le
  6. Jianfeng Gao
  7. Percy Liang
  8. Dacheng Tao

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrea Vedaldi
  8. Bernt Schiele

In Statistical and Nonlinear Physics

  1. Stefano Ermon
  2. Arash Vahdat
  3. Yang Song
  4. G. Em Karniadakis
  5. Jonathan Ho
  6. Karsten Kreis
  7. Jiaming Song
  8. Jun Zhu

In Model Architectures

  1. Yoshua Bengio
  2. Vladlen Koltun
  3. Aaron C. Courville
  4. David Kristjanson Duvenaud
  5. J. Kolter
  6. Christopher R'e
  7. Shaojie Bai
  8. Tri Dao

Citations in the corpus