Neural Networks with Cheap Differential Operators

By Ricky T. Q. Chen, David Kristjanson Duvenaud. 2019

39 citations recorded by the source corpus.

Gradients of neural networks can be computed efficiently for any architecture, but some applications require computing differential operators with higher time complexity. We describe a family of neural network architectures that allow easy access to a family of differential operators involving \emph{dimension-wise derivatives}, and we show how to modify the backward computation graph to compute them efficiently. We demonstrate the use of these operators for solving root-finding subproblems in implicit ODE solvers, exact density evaluation for continuous normalizing flows, and evaluating the Fokker-Planck equation for training stochastic differential equation models.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “Another approach relies on designing special dynamics functions, in which input dimensions are decoupled and then combine the detach-operator with one backpropagation pass to compute the divergence exactly (Chen & Duvenaud, 2019).”
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (2023, unknown): “However, the resulting setup departs from standard score matching and requires specialized and expensive network architectures (Chen & Duvenaud, 2019).”
Relative gradient optimization of the Jacobian term in unsupervised deep learning (2020, background): “[11] Tian Qi Chen and David K Duvenaud.”