Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy

By Jiehui Xu, Haixu Wu, Jianmin Wang, Mingsheng Long. International Conference on Learning Representations, 2021

1,096 citations recorded by the source corpus.

Unsupervised detection of anomaly points in time series is a challenging problem, which requires the model to derive a distinguishable criterion. Previous methods tackle the problem mainly through learning pointwise representation or pairwise association, however, neither is sufficient to reason about the intricate dynamics. Recently, Transformers have shown great power in unified modeling of pointwise representation and pairwise association, and we find that the self-attention weight distribution of each time point can embody rich association with the whole series. Our key observation is that due to the rarity of anomalies, it is extremely difficult to build nontrivial associations from abnormal points to the whole series, thereby, the anomalies' associations shall mainly concentrate on their adjacent time points. This adjacent-concentration bias implies an association-based criterion inherently distinguishable between normal and abnormal points, which we highlight through the \emph{Association Discrepancy}. Technically, we propose the \emph{Anomaly Transformer} with a new \emph{Anomaly-Attention} mechanism to compute the association discrepancy. A minimax strategy is devised to amplify the normal-abnormal distinguishability of the association discrepancy. The Anomaly Transformer achieves state-of-the-art results on six unsupervised time series anomaly detection benchmarks of three applications: service monitoring, space&earth exploration, and water treatment.

View Paper: https://www.semanticscholar.org/paper/a46b06a4b8b4deecf96a4e42cd19b4696f999e66

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Zhijian Xu
2. Tian Zhou
3. Ailing Zeng
4. Junchi Yan
5. Qingsong Wen
6. Qiang Xu
7. Ziqing Ma
8. Liang Sun

Citations in the corpus

• DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection (2023, methodology): “We compare our model with the 26 baselines for comprehensive evaluations, including the reconstruction-based model: AutoEncoder [61], LSTM-VAE [53], OmniAnomaly [66], BeatGAN [91], InterFusion [45], Anomaly Transformer [75]; the autoregressionbased models: VAR [4], Autoregression [58], LSTM-RNN [8],”
• Foundation Models for Time Series Analysis: A Tutorial and Survey (2024, unknown): “Taxonomy Standard TS Spatial TS Others (e.g., event) Jin et al. [47] Data ✔ ✔ ✘ Jiang et al. [44] Pipeline ✔ ✘ ✔ Zhang et al. [118] Pipeline ✔ ✘ ✘ Miller et al. [67] Pipeline ✔ ✘ ✘ (Ours) Methodology ✔ ✔ ✔ each requiring tailored approaches to effectively capture their inherent properties.”
• Deep Learning for Time Series Anomaly Detection: A Survey (2022, unknown): “3.4.1 Transformers.”
• FITS: Modeling Time Series with 10k Parameters (2023, methodology): “Following the previous works (Xu et al., 2022; Shen et al., 2020; Wu et al., 2023), we use Precision, Recall, and F1-score as metrics.”
• Transformers in Time Series: A Survey (2022, methodology): “AnomalyTrans [Xu et al., 2022] combines Transformer and Gaussian Prior-Association to make rare anomalies more distinguishable.”
• Transformers in Time-Series Analysis: A Tutorial (2022, methodology): “The BERT architecture has inspired many new Transformer models for time-series data [62, 65, 78, 63].”