Latent ODEs for Irregularly-Sampled Time Series

By Yulia Rubanova, Ricky T. Q. Chen, David Kristjanson Duvenaud. arXiv.org, 2019

340 citations recorded by the source corpus.

Time series with non-uniform intervals occur in many applications, and are difficult to model using standard recurrent neural networks (RNNs). We generalize RNNs to have continuous-time hidden dynamics defined by ordinary differential equations (ODEs), a model we call ODE-RNNs. Furthermore, we use ODE-RNNs to replace the recognition network of the recently-proposed Latent ODE model. Both ODE-RNNs and Latent ODEs can naturally handle arbitrary time gaps between observations, and can explicitly model the probability of observation times using Poisson processes. We show experimentally that these ODE-based models outperform their RNN-based counterparts on irregularly-sampled data.

Subfields: Electrical and Electronic Engineering, Management Science and Operations Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/25bb538adb125803a6f3f5632f31720d657adb13

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Management Science and Operations Research

In Signal Processing

Citations in the corpus

GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series (2019, methodology): “In a work submitted concomitantly to ours (Rubanova et al., 2019), the authors proposed a convincing new VAE architecture that uses a Neural-ODE architecture for both encoding and decoding the data.”
Universal Differential Equations for Scientific Machine Learning (2020, background): “An astute reader may recognize a neural ordinary differential equation u′ = NNθ(u, t) as the specific case of a onedimensional UDE defined in full by a neural network [25, 26, 27, 28, 29, 30].”
Liquid Time-constant Networks (2020, methodology): “n=5 We use the "Human Activity" dataset described in [47] in two distinct experimental settings.”
Neural Jump Stochastic Differential Equations (2019, methodology): “Most related to our approach is the recently proposed ODE-RNN model [32], which uses an RNN to make abrupt updates to a hidden state that mostly evolves continuous.”
Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs (2020, methodology): “Speciﬁcally, to build ﬂexible models for continuous-time, model-based RL, we ﬁrst introduce ways to incorporate action and time into the neural ODE work of [Chen et al., 2018, Rubanova et al., 2019].”
Long Expressive Memory for Sequence Modeling (2021, methodology): “…years with examples such as continuous-time neural ODEs (Chen et al., 2018; Queiruga et al., 2020; 2021) and their recurrent extensions ODE-RNNs (Rubanova et al., 2019), as well as RNNs based on discretizations of ODEs (Chang et al., 2018; Erichson et al., 2021; Chen et al., 2020; Lim et al.,…”
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network (2018, background): “Moreover, differential equations defined on the continuous time domain are a more natural fit for modeling certain real-life scenarios than the difference equations defined over the domain of evenly-discretized time intervals [6, 51].”
Multiscale modeling meets machine learning: What can we learn? (2019, background): “series containing measurements from the first 48 hours of a different patient’s admission to the intensive care unit [103].”
Scalable Gradients for Stochastic Differential Equations (2020, background): “This model family generalizes several existing families such as latent ODEs [11, 69] Gaussian state-space models [35, 78], and deep Kalman filters [39], and can naturally handle irregularly-sampled times series and missing observations.”