A comprehensive survey of anomaly detection techniques for high dimensional big data

By Srikanth Thudumu, Philip Branch, Jiong Jin, Jugdutt Singh. Journal of Big Data, 2020

408 citations recorded by the source corpus.

Anomaly detection in high dimensional data is becoming a fundamental research problem that has various applications in the real world. However, many existing anomaly detection techniques fail to retain sufficient accuracy due to so-called “big data” characterised by high-volume, and high-velocity data generated by variety of sources. This phenomenon of having both problems together can be referred to the “curse of big dimensionality,” that affect existing techniques in terms of both performance and accuracy. To address this gap and to understand the core problem, it is necessary to identify the unique challenges brought by the anomaly detection with both high dimensionality and big data problems. Hence, this survey aims to document the state of anomaly detection in high dimensional big data by representing the unique challenges using a triangular model of vertices: the problem (big dimensionality), techniques/algorithms (anomaly detection), and tools (big data applications/frameworks). Authors’ work that fall directly into any of the vertices or closely related to them are taken into consideration for review. Furthermore, the limitations of traditional approaches and current strategies of high dimensional data are discussed along with recent techniques and applications on big data required for the optimization of anomaly detection.

Subfields: Artificial Intelligence, Computer Networks and Communications

View Paper: https://www.semanticscholar.org/paper/aca6d5f3866372a4506cf15773ae298f18c3f453

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

A Comprehensive Survey on Graph Anomaly Detection With Deep Learning (2021, background): “There are some representative surveys on generalized anomaly detection techniques - [18], [45] and [44].”
TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data (2022, background): “Many data-driven industries, including ones related to distributed computing, Internet of Things (IoT), robotics and urban resource management [4, 46] are now adopting machine learning based unsupervised methods for anomaly detection.”
Machine Learning in Aerodynamic Shape Optimization (2022, background): “Unsupervised learning has prevailed in various important problems, such as anomaly detection [173, 174] and novelty detection [175, 176], clustering [177, 178], and dimensionality reduction[179, 180].”
Deep Learning for Medical Anomaly Detection – A Survey (2020, background): “One of the main challenges arises from the inconsistent behaviour of different anomalies, and the lack of constant definition of what constitutes an anomaly [1], [2], [3].”