GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series

By Edward De Brouwer, Jaak Simm, Adam Arany, Yves Moreau. Neural Information Processing Systems, 2019

404 citations recorded by the source corpus.

Modeling real-world multidimensional time series can be particularly challenging when these are sporadically observed (i.e., sampling is irregular both in time and across dimensions)-such as in the case of clinical patient data. To address these challenges, we propose (1) a continuous-time version of the Gated Recurrent Unit, building upon the recent Neural Ordinary Differential Equations (Chen et al., 2018), and (2) a Bayesian update network that processes the sporadic observations. We bring these two ideas together in our GRU-ODE-Bayes method. We then demonstrate that the proposed method encodes a continuity prior for the latent process and that it can exactly represent the Fokker-Planck dynamics of complex processes driven by a multidimensional stochastic differential equation. Additionally, empirical evaluation shows that our method outperforms the state of the art on both synthetic data and real-world data with applications in healthcare and climate forecast. What is more, the continuity prior is shown to be well suited for low number of samples settings.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/63c99c04869a52ac69850e21732b26d8633852ea

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs (2020, methodology): “Rubanova et al. [2019], De Brouwer et al. [2019] used a neural ODE to update the hidden state of recurrent cells; Chen et al. [2018] deﬁned latent variables for observations as the solution to an ODE; Kidger et al. [2020] adjusted trajectories based on subsequent observations with controlled…”
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers (2021, neutral): “LSSL LSSL-f CKConv UnICORNN N(C/R)DE ODE-RNN [45] GRU-ODE [16] 1→ 1 95.”
Multi-Time Attention Networks for Irregularly Sampled Time Series (2020, methodology): “De Brouwer et al. (2019) proposed GRU-ODE-Bayes, a continuous-time version of the Gated Recurrent Unit (Chung et al., 2014).”
Learning Differential Equations that are Easy to Solve (2020, background): “However, this approach has an O ( D 3 ) time cost. to De Brouwer et al. (2019) introduced a parameterization of neural ODEs analogous instantaneous Gated Recurrent Unit (GRU) recurrent neural network architectures in order to stabilize training dynamics.”
Efficiently Modeling Long Sequences with Structured State Spaces (2021, background): “As a continuous-time model, S4 automatically adapts to data sampled at diﬀerent rates, a challenging setting for time series with a dedicated line of work [10, 35, 37].”
Neural Controlled Differential Equations for Irregular Time Series (2020, background): “…input; GRU-D [6], which is essentially a GRU-∆ t model which applies a learnt exponential decay to the hidden state between observations; GRU-ODE [14, 25], which is an ODE analogous to the operation of a GRU and uses X as its input; ODE-RNN [13], which is a GRU-∆ t model which additionally…”
A Transformer-based Framework for Multivariate Time Series Representation Learning (2020, background): “There is a large variety of modeling approaches for univariate and multivariate time series, with deep learning models recently challenging and at times pushing the state of the art in tasks such as forecasting, regression and classification [7, 11, 30].”