Recurrent Neural Networks for Multivariate Time Series with Missing Values
By Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David A. Sontag, Yan Liu. 2016
2,465 citations recorded by the source corpus.
Multivariate time series data in practical applications, such as health care, geoscience, and biology, are characterized by a variety of missing values. In time series prediction and other related tasks, it has been noted that missing values and their missing patterns are often correlated with the target labels, a.k.a., informative missingness. There is very limited work on exploiting the missing patterns for effective imputation and improving prediction performance. In this paper, we develop novel deep learning models, namely GRU-D, as one of the early attempts. GRU-D is based on Gated Recurrent Unit (GRU), a state-of-the-art recurrent neural network. It takes two representations of missing patterns, i.e., masking and time interval, and effectively incorporates them into a deep model architecture so that it not only captures the long-term temporal dependencies in time series, but also utilizes the missing patterns to achieve better prediction results. Experiments of time series classification tasks on real-world clinical datasets (MIMIC-III, PhysioNet) and synthetic datasets demonstrate that our models achieve state-of-the-art performance and provide useful insights for better understanding and utilization of missing values in time series analysis.
Subfields: Artificial Intelligence, Health Information Management, Model Architectures, Medical & Health AI, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Ruslan Salakhutdinov
- Stefano Ermon
In Health Information Management
- Jimeng Sun
- Yuan Luo
- Cao Xiao
- Fei Wang
- Joseph D. Futoma
- Mihaela van der Schaar
- Edward Choi
- Phebe Vayanos
In Model Architectures
In Medical & Health AI
- Mihaela van der Schaar
- Jimeng Sun
- Fei Wang
- Cao Xiao
- Edward Choi
- Joseph D. Futoma
- Yuan Luo
- Phebe Vayanos
In Signal Processing
- Zachary Chase Lipton
- Yuan Luo
- Christopher R'e
- Joseph D. Futoma
- Albert Gu
- Karan Goel
- Ricky T. Q. Chen
- Yang Song
Citations in the corpus
- Neural Ordinary Differential Equations (2018, neutral): “Missing data can be addressed using generative time-series models (Álvarez and Lawrence, 2011; Futoma et al., 2017; Mei and Eisner, 2017; Soleimani et al., 2017a) or data imputation (Che et al., 2018).”
- Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow (2018, background): “Recurrent neural networks are popular for control tasks [18], automatic video captioning [31], or generic sequence predictions [7].”
- Recurrent Neural Networks for Classifying Relations in Clinical Notes (2017, positive): “[40] applied GRUs to perform mortality and diagnosis code prediction using time series data consisting of physiologic measurements, labtests values, and prescriptions.”
- GRAM: Graph-based Attention Model for Healthcare Representation Learning (2016, background): “3098126 networks (RNN) [7, 9, 10, 25], convolutional neural networks (CNN) [30] or stacked denoising autoencoders (SDA) [6, 29], demonstrating signi cant performance enhancement for diverse prediction tasks.”
- Scalable and accurate deep learning with electronic health records (2018, background): “eling the temporal sequence of events that occurred in a patient’s record, which may enhance accuracy in scenarios that depend on the order of events, with convolutional and recurrent neural networks [61, 2, 17, 81, 16, 92]. In general, prior work has focused on a subset of features available in the”
- HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, methodology): “We note that the setting of missing data is a topic of independent interest and we compare against SOTA methods, including the GRU-D [11] which learns a decay between observations, and neural ODE methods which models segments between observations with an ODE.”
- A Review of Challenges and Opportunities in Machine Learning for Health. (2018, background): “…memory models are needed for sequence prediction tasks, because such records are not evenly spaced, can cover very long durations, and early events can affect patient state many years later; early work in this space has targeted learning individual variable decays in EHR data with missingness 107 .”
- Evolutionary neural AutoML for deep learning (2019, background): “ds of millions of parameters, requiring large computational resources to train and run. They are also highly complex, and their performance depends on their architecture and choice of hyperparameters [13, 23, 39]. Much of the recent research in deep learning indeed focuses on discovering specialized”
- Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks (2019, methodology): “To illustrate the effects of non-stationarity, we benchmark four commonly used models in the machine learning literature for clinical prediction from time series data: logistic regression (LR), random forests (RF), long short-term memory (LSTM) network (Hochreiter and Schmidhuber, 1997) and a gated ”
- Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks (2017, background): “Recurrent neural networks (RNNs) can be used for modeling multivariate time series data in healthcare with missing values [6, 18].”
- A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection (2023, background): “While some deep neural network-based works, such as [173] and [174], have mitigated these limitations, they do not explicitly consider inter-variable dependencies.”
- Stacked Bidirectional and Unidirectional LSTM Recurrent Neural Network for Forecasting Network-wide Traffic State with Missing Values (2020, methodology): “However, solving the imputation and prediction tasks at the same time often leads to a two-step process where imputation and prediction models are separated Che et al. (2018).”
- Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets (2017, background): “With the goal of accurately predicting these clinical outcomes, researchers have developed novel machine learning models [1, 2] and scoring systems [3] while measuring the”
- CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation (2021, methodology): “A popular approach using deep learning is to use RNNs, including LSTMs and GRUs, for sequence modeling [17, 8, 7].”