Attend and Diagnose: Clinical Time Series Analysis using Attention Models

By Huan-Zhi Song, Deepta Rajan, Jayaraman J. Thiagarajan, Andreas S. Spanias. 2017

530 citations recorded by the source corpus.

With widespread adoption of electronic health records, there is an increased emphasis for predictive models that can effectively deal with clinical time-series data. Powered by Recurrent Neural Network (RNN) architectures with Long Short-Term Memory (LSTM) units, deep neural networks have achieved state-of-the-art results in several clinical prediction tasks. Despite the success of RNN, its sequential nature prohibits parallelized computing, thus making it inefficient particularly when processing long sequences. Recently, architectures which are based solely on attention mechanisms have shown remarkable success in transduction tasks in NLP, while being computationally superior. In this paper, for the first time, we utilize attention models for clinical time-series modeling, thereby dispensing recurrence entirely. We develop the SAnD (Simply Attend and Diagnose) architecture, which employs a masked, self-attention mechanism, and uses positional encoding and dense interpolation strategies for incorporating temporal order. Furthermore, we develop a multi-task variant of SAnD to jointly infer models with multiple diagnosis tasks. Using the recent MIMIC-III benchmark datasets, we demonstrate that the proposed approach achieves state-of-the-art performance in all tasks, outperforming LSTM models and classical baselines with hand-engineered features.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting (2019, methodology): “Despite performance gains over LSTM-based iterative methods, interpretability is still not straightforward with standard direct methods.”
Secure and Robust Machine Learning for Healthcare: A Survey (2020, unknown): “) by integrating clinical notes with multivariate and time-series measurements data [76].”
Multitask learning and benchmarking with clinical time series data (2017, methodology): “ensation prediction [44], length-of-stay prediction [42, 44, 45], phenotyping [38, 39, 46] and readmission prediction [47]. Additionally, attention-based RNNs were applied for all our benchmark tasks [48]. In a parallel work [49], another set of benchmark tasks based on MIMIC-III was introduced that”
Multi-Time Attention Networks for Irregularly Sampled Time Series (2020, background): “Several recent approaches have also used attention mechanism to model irregularly sampled time series (Song et al., 2018; Tan et al., 2020; Zhang et al., 2019) as well as medical concepts (Peng et al.”
Deep Learning for Time Series Forecasting: Tutorial and Literature Survey (2020, methodology): “SAnD [173] MLP + Attention Point 1 l2 , cross-entropy, multi-label classification loss AUROC, MASE, MSE clinical Transformer [116] based model adjusted for time series forecasting”
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2020, methodology): “Transformer-based time-series model The most related works (Song et al. 2018; Ma et al. 2019; Li et al. 2019) all start from a trail on applying Transformer in time-series data and fail in LSTF forecasting as they use the vanilla Transformer.”
Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformer (2019, methodology): “There are recent works that use Transformer on medical records (Song et al. 2018; Wang and Xu 2019; Shang et al. 2019a; Li et al. 2019), but they either simply replace RNN with Transformer to handle ICU records, or directly apply BERT learning objective on medical records, and do not utilize the…”
Learning Dynamics and Heterogeneity of Spatial-Temporal Graph Data for Traffic Forecasting (2021, background): “SAnD [43], DSANet [44] extend its usage into multivariate time series prediction.”
Deep Learning for Anomaly Detection in Time-Series Data: Review, Analysis, and Guidelines (2021, methodology): “Recently, Transformer with a stacked encoder-decoder structure, which consists only of attention mechanisms, is employed in several works [73], [109], [119], [120].”
Deep Learning for Time Series Anomaly Detection: A Survey (2022, unknown): “The SAnD (Simply Attend and Diagnose) [160] uses attention models to model clinical time series, eliminating the need for recurrence.”
A General Survey on Attention Mechanisms in Deep Learning (2022, background): “Attention can generally also be applied to many problems that use a time series as input, be it medical [29], financial [30], or anything else, as long as feature vectors can be extracted.”
Attention in Natural Language Processing (2019, background): “recommendation [22], [23], time-series analysis [24], [25], games [26], and mathematical problems [27], [28].”
Transformers in Time-Series Analysis: A Tutorial (2022, methodology): “Song et al. introduced Transformers to predict various variables of clinical interest from the MIMIC-III benchmark dataset [80], named Simply Attend and Diagnose (SAnD) [60].”
Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting (2021, background): “DeepAR [34] combines autoregressive methods and RNNs to model the probabilistic distribution of future series.”