Learning Differential Equations that are Easy to Solve
By Jacob Kelly, Jesse Bettencourt, Matthew J. Johnson, David Kristjanson Duvenaud. 2020
131 citations recorded by the source corpus.
Differential equations parameterized by neural networks become expensive to solve numerically as training progresses. We propose a remedy that encourages learned dynamics to be easier to solve. Specifically, we introduce a differentiable surrogate for the time cost of standard numerical solvers, using higher-order derivatives of solution trajectories. These derivatives are efficient to compute with Taylor-mode automatic differentiation. Optimizing this additional objective trades model performance against the time cost of solving the learned dynamics. We demonstrate our approach by training substantially faster, while nearly as accurate, models in supervised classification, density estimation, and time-series modelling tasks.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Benjamin Paul Chamberlain
- Emanuele Rossi
- James Rowbottom
- Michael M. Bronstein
- Delu Zeng
- Wei Chen
- Shian Du
- Jian Xu
In Statistical and Nonlinear Physics
- Benjamin Paul Chamberlain
- James Rowbottom
- Michael M. Bronstein
- Delu Zeng
- Emanuele Rossi
- Wei Chen
- Shian Du
- Jian Xu
Citations in the corpus
- Flow Matching for Generative Modeling (2022, neutral): “…regularize the ODE to be easier to solve, e.g. , using augmentation (Dupont et al., 2019), adding regularization terms (Yang & Karniadakis, 2019; Finlay et al., 2020; Onken et al., 2021; Tong et al., 2020; Kelly et al., 2020), or stochastically sampling the integration interval (Du et al., 2022).”
- TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed (2022, methodology): “Motivated by the similarities between training CNFs and solving OT problems [63, 64], some previous works regularize the minimization problem (5) to enforce a straight trajectory and significantly faster training [37, 60, 61].”
- GRAND: Graph Neural Diffusion (2021, methodology): “For the larger datasets, we used kinetic energy and Jacobian regularization (Finlay et al., 2020; Kelly et al., 2020).”