Time-Series Anomaly Detection Service at Microsoft

By Hansheng Ren, Bixiong Xu, Yujing Wang, Chao Yi, Congrui Huang, Xiaoyu Kou, T. Xing, Mao Yang, Jie Tong, Qi Zhang. Knowledge Discovery and Data Mining, 2019

651 citations recorded by the source corpus.

Large companies need to monitor various metrics (for example, Page Views and Revenue) of their applications and services in real time. At Microsoft, we develop a time-series anomaly detection service which helps customers to monitor the time-series continuously and alert for potential incidents on time. In this paper, we introduce the pipeline and algorithm of our anomaly detection service, which is designed to be accurate, efficient and general. The pipeline consists of three major modules, including data ingestion, experimentation platform and online compute. To tackle the problem of time-series anomaly detection, we propose a novel algorithm based on Spectral Residual (SR) and Convolutional Neural Network (CNN). Our work is the first attempt to borrow the SR model from visual saliency detection domain to time-series anomaly detection. Moreover, we innovatively combine SR and CNN together to improve the performance of SR model. Our approach achieves superior experimental results compared with state-of-the-art baselines on both public datasets and Microsoft production data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/25d4c67fac5350d80c059c7e8bf09b988b54e02f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jules Samaran
2. Defu Cao
3. Bixiong Xu
4. Astha Garg
5. V. Varadharajan
6. Suhuai Luo
7. Ce Zhang
8. Dan Pei

In Computer Vision and Pattern Recognition

1. Jules Samaran
2. Bixiong Xu
3. Astha Garg
4. V. Varadharajan
5. Suhuai Luo
6. Ce Zhang
7. Jie Tong
8. Juanyong Duan

Citations in the corpus

• An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series (2021, background): “We did not evaluate some recently proposed algorithms as their source-code is proprietary and is not straightforward to implement [11], [14].”
• Multivariate Time Series Anomaly Detection and Interpretation using Hierarchical Inter-Metric and Temporal Embedding (2021, methodology): “We choose the sliding window ( x t − W + 1 , . . . , x t ) as the input for detecting anomaly at time t , and use the score for the last data x t in the window as the anomaly score (following [14, 23, 30, 33]).”
• DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection (2023, methodology): “Time series anomaly detection is widely used in real-world applications, including but not limited to industrial equipment status monitoring, financial fraud detection, fault diagnosis, and daily monitoring and maintenance of automobiles [3, 20, 27, 54, 75].”
• Multivariate Time-series Anomaly Detection via Graph Attention Network (2020, methodology): “Recent advances in neural networks also lay a strong foundation for time-series anomaly detection [1], [8], [21].”
• Deep Learning for Time Series Anomaly Detection: A Survey (2022, neutral): “Manuscript submitted to ACM”
• Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting (2020, background): “For instance, State Frequency Memory (SFM) network [32] combines the advantages of DFT and LSTM jointly for stock price prediction; Spectral Residual (SR) model [23] leverages DFT and achieves state-of-the-art performances in”
• TS2Vec: Towards Universal Representation of Time Series (2021, methodology): “Hence, we follow exactly the same evaluation strategy as Ren et al. (2019); Xu et al. (2018), where anomalies detected within a certain delay (7 steps for minutely data and 3 steps for hourly data) are considered correct.”
• A Survey on Machine Learning Techniques for Cyber Security in the Last Decade (2020, methodology): “CNN is widely being used in image recognition [242], drug discovery [243], and anomaly detection [244], [245], to name a few.”
• Anomaly Detection in Time Series: A Comprehensive Evaluation (2022, methodology): “Time series classification, in contrast, describes the process of assigning entire time series to certain classes; it is often used as a post-processing step to classify detected anomalies into domain-specific classes, but some anomaly detection algorithms, such as PS-SVM [85], SR-CNN [112], COPOD [”