A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder

By Daehyung Park, Yuuna Hoshi, Charles C. Kemp. IEEE Robotics and Automation Letters, 2017

1,066 citations recorded by the source corpus.

The detection of anomalous executions is valuable for reducing potential hazards in assistive manipulation. Multimodal sensory signals can be helpful for detecting a wide range of anomalies. However, the fusion of high-dimensional and heterogeneous modalities is a challenging problem for model-based anomaly detection. We introduce a long short-term memory-based variational autoencoder (LSTM-VAE) that fuses signals and reconstructs their expected distribution by introducing a progress-based varying prior. Our LSTM-VAE-based detector reports an anomaly when a reconstruction-based anomaly score is higher than a state-based threshold. For evaluations with 1555 robot-assisted feeding executions, including 12 representative types of anomalies, our detector had a higher area under the receiver operating characteristic curve of 0.8710 than 5 other baseline detectors from the literature. We also show the variational autoencoding and state-based thresholding are effective in detecting anomalies from 17 raw sensory signals without significant feature engineering effort.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/bc9293bcee13cae5cff8a088f4038c4236decd42

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data (2022, methodology): “The Omnianomaly [45] uses a stochastic recurrent neural network (similar to an LSTM-Variational Autoencoder [39]) and a planar normalizing flow to generate reconstruction probabilities.”
Learning Graph Structures With Transformer for Multivariate Time-Series Anomaly Detection in IoT (2021, methodology): “Moreover, we have the following observations: (1) Compared to the conventional unsupervised approaches such as PCA, KNN, FB, deep learning-based techniques (AE, LSTM-VAE, MAD-GAN, etc.) generally have a better detection performance on both datasets.”
Timeseries Anomaly Detection using Temporal Hierarchical One-Class Network (2020, neutral): “These include (i) the 1.0 encoder-decoder scheme for anomaly detection (EncDec-AD) [17], (ii) LSTM-VAE [21], (iii) MAD-GAN [13], (iv) BeatGAN [33], (v) OmniAnomaly [28], and (vi) multi-scale convolutional recurrent encoder-decoder (MSCRED) [31].”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, methodology): “VAEs have been used in various forms for AD [276], [278], [394], for instance, on multimodal sequential data with LSTMs in robot-assisted feeding [395] and for new physics mining at the Large Hadron Collider [74].”
A Survey on Anomaly Detection for Technical Systems using LSTM Networks (2021, methodology): “In [29], a variational LSTM AE is introduced for anomaly detection purposes.”
Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network (2019, methodology): “To demonstrate the effectiveness of OmniAnomaly, we first compare it with four state-of-the-art unsupervised approaches for multivariate time series anomaly detection: LSTM with nonparametric dynamic thresholding (LSTM-NDT for short) [6], EncDec-AD [11], DAGMM [27], and LSTM-VAE [16], which have bee”
DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection (2023, methodology): “Such methods can be dated back to AutoEncoder [61], LSTM-VAE [53], Spectral Residual (SR) [57], RobustTAD [26] and so on.”
Graph Neural Network-Based Anomaly Detection in Multivariate Time Series (2021, methodology): “…Network (CNN) based models (Munir et al. 2018), Long Short Term Memory (LSTM) (Filonov, Lavrentyev, and Vorontsov 2016; Hund-man et al. 2018a; Park, Hoshi, and Kemp 2018) and Generative Adversarial Networks (GAN) models (Zhou et al. 2019; Li et al. 2019), have found success in practical time…”
Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture (2019, methodology): “Recently, the VAE has been used for unsupervised reconstruction based anomaly detection by applying a reconstruction error as an anomaly score [39].”
Anomaly Detection in Time Series: A Comprehensive Evaluation (2022, neutral): “…TCN-AE[135] AD-LTI [148] LSTM-AD [89] AE [117] VELC[158] Bagel [79] HealthESN [32] MTAD-GAT [161] MoteESN[30] RADM[40] MSCRED [159] LSTM-VAE [106] OmniAnomaly [125] OceanWNN [143] Image-embedding-CAE [44] Donut [150] MAD-GAN[77] TAnoGAN [8] PAD[33] Torsk [60] SSA [155] NumentaHTM [3]…”
Multivariate Time Series Anomaly Detection and Interpretation using Hierarchical Inter-Metric and Temporal Embedding (2021, methodology): “…has been an active research topic in the SIGKDD community these years [2, 14, 30], and is widely used to monitor the status ( i.e. , metrics) of entities ( e.g. , systems, services) in the application domain of manufacturing industry and Information Technology (IT) systems [14, 19, 20, 22, 30, 34].”
Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy (2021, methodology): “Park et al. (2018) presented the LSTM-VAE model that employed the LSTM backbone for temporal modeling and the Variational AutoEncoder (VAE) for reconstruction.”
A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection (2023, methodology): “For example, MTAD-GAT [35] and LSTM-VAE [139] both use a variational objective [148], but MTAD-GAT employs a graph attention network as a spatial-temporal encoder to learn inter-variable and inter-temporal dependencies.”
Deep Learning for Time Series Anomaly Detection: A Survey (2022, neutral): “2) LSTM-VAE [140] T Semi P OmniAnomaly [162] T Un W ✓ Point + Subseq ✓ STORN [158] ST Un P Point GGM-VAE [73] T Un W Subseq SISVAE [111] T Un W Point VAE-GAN [135] T Semi W Point VELC [188] T Un TopoMAD [78] ST Un W Subseq ✓ PAD [31] T Un W Subseq InterFusion [116] ST Un W ✓ Subseq MT-RVAE* [177] ST”
Deep Learning for Anomaly Detection in Time-Series Data: Review, Analysis, and Guidelines (2021, methodology): “…in multivariate time series anomaly detection, categorized as follows : • AE: DAGMM [103], MSCRED [51], OmniAnomaly [68] • VAE: LSTM-VAE [112], USAD [72] • GAN: MAD-GAN [59] • RNN: THOC [60] • Transformer: GTA [109] • GNN: GTA [109], GDN [108] Table 6 shows the anomaly detection…”
An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series (2021, methodology): “Model DMDS MSL SKAB SMAP SMD SWaT WADI Mean Avg Rank Raw Signal 0.7073 0.4379 0.4855 0.4953 0.7475 0.7273 0.7733 0.6249 10.0 PCA 0.7413 0.6175 0.5037 0.5684 0.7900 0.8249 0.6583 0.6720 7.3 UAE 0.8852 0.6712 0.5088 0.6256 0.8298 0.8279 0.7405 0.7270 2.6 FC AE 0.9280 0.6602 0.4914 0.6272 0.8028 0.8085”
Multivariate Time-series Anomaly Detection via Graph Attention Network (2020, background): “GAN-Li [10], MAD-GAN [11] and LSTM-VAE [12].”
Taming VAEs (2018, background): “Deep variational auto-encoders (VAEs) have made substantial progress in the last few years on modelling complex data such as images [1, 2, 3, 4, 5, 6, 7], speech synthesis [8, 9], molecular discovery [10, 11, 12], robotics [13, 14, 15] and 3d-scene understanding [16, 17, 18].”