Evaluating Real-Time Anomaly Detection Algorithms -- The Numenta Anomaly Benchmark

By Alexander Lavin, Subutai Ahmad. International Conference on Machine Learning and Applications, 2015

507 citations recorded by the source corpus.

Much of the world's data is streaming, time-series data, where anomalies give significant information in critical situations, examples abound in domains such as finance, IT, security, medical, and energy. Yet detecting anomalies in streaming data is a difficult task, requiring detectors to process data in real-time, not batches, and learn while simultaneously making predictions. There are no benchmarks to adequately test and score the efficacy of real-time anomaly detectors. Here we propose the Numenta Anomaly Benchmark (NAB), which attempts to provide a controlled and repeatable environment of open-source tools to test and measure anomaly detection algorithms on streaming data. The perfect detector would detect all anomalies as soon as possible, trigger no false alarms, work with real-world time-series data across a variety of domains, and automatically adapt to changing statistics. Rewarding these characteristics is formalized in NAB, using a scoring algorithm designed for streaming data. NAB evaluates detectors on a benchmark dataset with labeled, real-world time-series data. We present these components, and give results and analyses for several open source, commercially-used algorithms. The goal for NAB is to provide a standard, open source framework with which the research community can compare and evaluate different algorithms for detecting anomalies in streaming data.

Subfields: Artificial Intelligence, Computer Networks and Communications, Epidemiology

View Paper: https://www.semanticscholar.org/paper/08f886674d0a0a9d8eb87d3c4f57a31c4f5d132e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Epidemiology

Citations in the corpus

ADBench: Anomaly Detection Benchmark (2022, background): “Also, “anomaly detection” is an overloaded term; there are AD benchmarks for time-series [85, 87, 132], graph [101], CV [6, 27, 203] and NLP [143], but they are different from tabular AD in nature.”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, background): “For time-series AD [169], [195], [201]–[204], where forecasting (i.”
40 years of cognitive architectures: core cognitive abilities and practical applications (2018, methodology): “This kind of learning is frequently performed to obtain implicit knowledge about the environment, such as spatial maps [RCS (Schlenoff et al. 2005), AIS (Hayes-Roth et al. 1993), MicroPsi (Bach et al. 2007)], clustering visual features [HTM (Kostavelis et al. 2012), BECCA (Rohrer 2011b), Leabra (O’R”
Progress in Outlier Detection Techniques: A Survey (2019, background): “[230] Numenta Anomaly Benchmark (NAB) framework, there is a lack of”
Unsupervised real-time anomaly detection for streaming data (2017, methodology): “We briefly describe each of these below (full details can be ound in [2] ).”
Data-Driven Blood Glucose Pattern Classification and Anomalies Detection: Machine-Learning Applications in Type 1 Diabetes (2019, background): “Hawkins defined anomalies as “observations that deviate much from the other observations so as to arouse suspicions that it could be generated by a different process” [7,17,18].”
Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications (2018, background): “, [20], but most are not widely accepted, likely because they are too complicated.”
Deep Learning for Time Series Anomaly Detection: A Survey (2022, unknown): “Instead of using VAE in Donut, Bagel employs conditional variational autoencoder (CVAE) [107], [108] and considers temporal information.”
DeepAnT: A Deep Learning Approach for Unsupervised Anomaly Detection in Time Series (2019, methodology): “To highlight the generic nature of the proposed technique, we tested it on real as well as synthetic datasets from different domains including road traffic [38], network utilization [38], on-line advertisement [38], internet traffic [55], space shuttle [58], and health [53].”
Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding (2018, methodology): “omalous region, a false positive is recorded. For simplicity, we do not make scoring adjustments based on how early an anomaly was detected or the distance between false positives and labeled regions [27]. Batch processing. Telemetry values are aggregated into one minute windows and evaluated in bat”
Smart Anomaly Detection in Sensor Systems (2020, background): “The Numenta Anomaly Benchmark (NAB) [31], [32] offers time series datasets concerned with real-time anomaly detection in streaming data generated by sensor systems.”