Neural Controlled Differential Equations for Irregular Time Series

By Patrick Kidger, James Morrill, James Foster, Terry Lyons. Neural Information Processing Systems, 2020

758 citations recorded by the source corpus.

Neural ordinary differential equations are an attractive option for modelling temporal dynamics. However, a fundamental issue is that the solution to an ordinary differential equation is determined by its initial condition, and there is no mechanism for adjusting the trajectory based on subsequent observations. Here, we demonstrate how this may be resolved through the well-understood mathematics of \emph{controlled differential equations}. The resulting \emph{neural controlled differential equation} model is directly applicable to the general setting of partially-observed irregularly-sampled multivariate time series, and (unlike previous work on this problem) it may utilise memory-efficient adjoint-based backpropagation even across observations. We demonstrate that our model achieves state-of-the-art performance against similar (ODE or RNN based) models in empirical studies on a range of datasets. Finally we provide theoretical results demonstrating universal approximation, and that our model subsumes alternative ODE models.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Signal Processing

View Paper: https://www.semanticscholar.org/paper/31603b3339f4da5bdc6b7de4231bd1ddfb32a50a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Signal Processing

Citations in the corpus

HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, background): “[40], we consider the Character Trajectories dataset [4], where the goal is to classify a character from a sequence of pen stroke measurements, collected from one user at a fixed sampling rate.”
A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection (2023, background): “STG-NCDE [75] shares a similar idea but operates under the framework of neural controlled differential equations (NCDEs) [129].”
Universal Differential Equations for Scientific Machine Learning (2020, background): “An astute reader may recognize a neural ordinary differential equation u′ = NNθ(u, t) as the specific case of a onedimensional UDE defined in full by a neural network [25, 26, 27, 28, 29, 30].”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, methodology): “, 2020), with methods developed for prediction and control problems (Kidger et al., 2020; Massaroli et al., 2021), as well as acceleration of numerical schemes (Poli et al.”
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers (2021, methodology): “We highlight our primary baselines, three very recent works explicitly designed for long sequences: CKConv (a continuous-time CNN) [44], UnICORNN (an ODE-inspired RNN) [47], and Neural Controlled/Rough Diﬀerential Equations (NCDE/NRDE) (a sophisticated NDE) [31, 37].”