Robust log-based anomaly detection on unstable log data

By Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xinsheng Yang, Qian Cheng, Ze Li and 7 more. ESEC/SIGSOFT FSE, 2019

766 citations recorded by the source corpus.

Logs are widely used by large and complex software-intensive systems for troubleshooting. There have been a lot of studies on log-based anomaly detection. To detect the anomalies, the existing methods mainly construct a detection model using log event data extracted from historical logs. However, we find that the existing methods do not work well in practice. These methods have the close-world assumption, which assumes that the log data is stable over time and the set of distinct log events is known. However, our empirical study shows that in practice, log data often contains previously unseen log events or log sequences. The instability of log data comes from two sources: 1) the evolution of logging statements, and 2) the processing noise in log data. In this paper, we propose a new log-based anomaly detection approach, called LogRobust. LogRobust extracts semantic information of log events and represents them as semantic vectors. It then detects anomalies by utilizing an attention-based Bi-LSTM model, which has the ability to capture the contextual information in the log sequences and automatically learn the importance of different log events. In this way, LogRobust is able to identify and handle unstable log events and sequences. We have evaluated LogRobust using logs collected from the Hadoop system and an actual online service system of Microsoft. The experimental results show that the proposed approach can well address the problem of log instability and achieve accurate and robust results on real-world, ever-changing log data.

Subfields: Artificial Intelligence, Computer Networks and Communications

View Paper: https://www.semanticscholar.org/paper/848cf00ab64b02975904d28f8c3a6c9416261465

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hongyu Zhang
2. Van-Hoang Le
3. Michael R. Lyu
4. Shilin He
5. Pinjia He
6. Charu C. Aggarwal
7. Burr Settles

In Computer Networks and Communications

1. Hongyu Zhang
2. Van-Hoang Le
3. Michael R. Lyu
4. Yuxin Su
5. Shilin He
6. Pinjia He
7. Tianyi Yang
8. Zhuangbin Chen

Citations in the corpus

• A Survey on Automated Log Analysis for Reliability Engineering (2020, methodology): “Poorly written static text and inconsistent dynamic content could affect the subsequent diagnosis and maintenance activities [86, 111, 136, 196].”
• Log-based Anomaly Detection Without Log Parsing (2021, methodology): “In this section, we evaluate the impact of log parsing errors on two representative anomaly detection methods SVM-based method [30], and LogRobust [8].”
• LogBERT: Log Anomaly Detection via BERT (2021, unknown): “Recently, many deep learning-based log anomaly detection approaches are proposed for log anomaly detection [21,2,22,23,8,17].”