Detecting large-scale system problems by mining console logs

By Wei Xu, Ling Huang, Armando Fox, David A. Patterson, Michael I. Jordan. Symposium on Operating Systems Principles, 2009

1,360 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/c1e25bb0c3d0c3a8c3b2bf8e9735a9aadc175b45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

1. Hongyu Zhang
2. Van-Hoang Le
3. Michael R. Lyu
4. Pinjia He
5. Zibin Zheng
6. Jieming Zhu
7. Shilin He
8. Min Du

In Information Systems

1. Hongyu Zhang
2. Van-Hoang Le
3. Michael R. Lyu
4. Zibin Zheng
5. Min Du
6. Feifei Li
7. Pinjia He
8. Shilin He

In Software

1. Hongyu Zhang
2. Min Du
3. Feifei Li
4. Michael R. Lyu
5. Pinjia He
6. Shilin He
7. Van-Hoang Le
8. Jian-Guang Lou

Citations in the corpus

• Understanding and detecting real-world performance bugs (2012, background): “Many previous works tried to isolate bottlenecks in (semi-) blackbox systems [1], correlate system metrics with performance behaviors [8, 48], avoid performance problems at runtime [52], and diagnose problems through logs [59].”
• Developing a predictive model of quality of experience for internet video (2013, background): “Decision trees can be directly mapped into event processing rules that system designers are typically familiar with [39].”
• DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning (2017, methodology): “Principal Component Analysis (PCA) method [39] assumes that there are different “sessions” in a log file that can be easily identified by a session id attached to each log entry.”
• LogBERT: Log Anomaly Detection via BERT (2021, neutral): “Regarding baselines, we leverage the package Loglizer [4] to evaluate PCA, OCSVM, iForest as well as LogCluster for anomaly detection and adopt the open source deep learning-based log analysis toolkit to evaluate DeepLog and LogAnomaly 3 .”
• Tools and Benchmarks for Automated Log Parsing (2018, methodology): “Typical applications include anomaly detection [12], [18], [23], problem diagnosis [4], [5], program verification [55], [56], performance modeling [3], etc.”
• A Survey on Automated Log Analysis for Reliability Engineering (2020, methodology): “en-source implementation of existing log parsers, this survey focuses on the methodological comparison of these parsers. In additional, while parsing rules can be constructed by analyzing source code [144], we focus on parsers that only utilize logs as input. Mode. Mode is the most important log par”
• Experience Report: System Log Analysis for Anomaly Detection (2016, methodology): “Fortunately, by exploring an abundance of literature and intensively contacting the corresponding authors, we have successfully obtained two log datasets, HDFS data [47] and BGL data [36], which are suitable for evaluating existing anomaly detection methods.”
• Robust Anomaly Detection and Backdoor Attack Detection Via Differential Privacy (2019, methodology): “Over the past decade this log dataset has been extensively used for research in system log anomaly detection ([15, 16, 5]).”
• Mining Invariants from Console Logs for System Problem Detection (2010, background): “Compared with the state of art, our approach can not only detect numerous real problems with high accuracy but also provide intuitive insight into the problems.”
• Structured Comparative Analysis of Systems Logs to Diagnose Performance Problems (2012, methodology): “[32] have considered the automatic matching of log statements to source code, which requires tight coupling with programming languages to construct abstract syntax trees.”
• Spell: Streaming Parsing of System Event Logs (2016, methodology): “To alleviate the pain of diving into massive unstructured log data, in most prior work, the first and foremost step is to automatically parse the unstructured system logs to structured data [2], [3], [4], [6].”
• Log-based Anomaly Detection Without Log Parsing (2021, methodology): “HDFS dataset [34], [7] contains 11,175,629 log messages collected from a Hadoop Distributed File System on the Amazon EC2 platform.”
• Carat: collaborative energy diagnosis for mobile devices (2013, background): “Recent work shows how access to source code can facilitate tasks like log analysis [43] and distributed diagnosis [12].”
• The Statistics of Streaming Sparse Regression (2014, background): “In many areas such as astrophysics [1, 6], environmental sensor networks [42], distributed computer systems diagnostics [61], and advertisement click prediction [36], a system generates a high-throughput stream of data in real-time.”
• Robust log-based anomaly detection on unstable log data (2019, methodology): “Over the years, many automated log-based approaches have been proposed to detect system anomalies [3, 10, 18, 29, 44].”
• Log Clustering Based Problem Identification for Online Service Systems (2016, methodology): “To facilitate analysis, a common practice is to parse the log messages into constant strings and parameters [6, 11, 12, 24] and form abstract log messages.”