LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

By Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun and 1 more. International Joint Conference on Artificial Intelligence, 2019

733 citations recorded by the source corpus.

Recording runtime status via logs is common for almost every computer system, and detecting anomalies in logs is crucial for timely identifying malfunctions of systems. However, manually detecting anomalies for logs is time-consuming, error-prone, and infeasible. Existing automatic log anomaly detection approaches, using indexes rather than semantics of log templates, tend to cause false alarms. In this work, we propose LogAnomaly, a framework to model unstructured a log stream as a natural language sequence. Empowered by template2vec, a novel, simple yet effective method to extract the semantic information hidden in log templates, LogAnomaly can detect both sequential and quantitive log anomalies simultaneously, which were not done by any previous work. Moreover, LogAnomaly can avoid the false alarms caused by the newly appearing log templates between periodic model retrainings. Our evaluation on two public production log datasets show that LogAnomaly outperforms existing log-based anomaly detection methods.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/b2cf224e2d6e65c72b7ee3f06c54b0230909d597

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hongyu Zhang
2. Van-Hoang Le
3. Shuhan Yuan
4. Michael R. Lyu
5. Haixuan Guo
6. Shilin He
7. Pinjia He
8. Xintao Wu

In Computer Networks and Communications

1. Hongyu Zhang
2. Van-Hoang Le
3. Haixuan Guo
4. Shuhan Yuan
5. Xintao Wu
6. Michael R. Lyu
7. Yuxin Su
8. Shilin He

In Information Systems

1. Hongyu Zhang
2. Van-Hoang Le
3. Xintao Wu
4. Haixuan Guo
5. Shuhan Yuan
6. Michael R. Lyu
7. Yuxin Su
8. Shilin He

Citations in the corpus

• Log-based Anomaly Detection Without Log Parsing (2021, methodology): “However, LogAnomaly [12] has no publicly available implementation and requires operators’ domain knowledge (to manually add domain-specific synonyms and antonyms).”
• A Survey on Automated Log Analysis for Reliability Engineering (2020, background): “Poorly written static text and inconsistent dynamic content could affect the subsequent diagnosis and maintenance activities [86, 111, 136, 196].”
• LogBERT: Log Anomaly Detection via BERT (2021, background): “Most of the existing approaches adopt recurrent neural networks, especially long-short term memory (LSTM) or gated recurrent unit (GRU) to model the normal log key sequences and derive anomalous scores to detect the anomalous log sequences [2,23,17].”