Navigating the Data Lake with DATAMARAN: Automatically Extracting Structure from Log Datasets

By Yihan Gao, Silu Huang, Aditya G. Parameswaran. 2017

56 citations recorded by the source corpus.

Organizations routinely accumulate semi-structured log datasets generated as the output of code; these datasets remain unused and uninterpreted, and occupy wasted space---this phenomenon has been colloquially referred to as "data lake'' problem. One approach to leverage these semi-structured datasets is to convert them into a structured relational format, following which they can be analyzed in conjunction with other datasets. We present DATAMARAN, an tool that extracts structure from semi-structured log datasets with no human supervision. DATAMARAN automatically identifies field and record endpoints, separates the structured parts from the unstructured noise or formatting, and can tease apart multiple structures from within a dataset, in order to efficiently extract structured relational datasets from semi-structured log datasets, at scale with high accuracy. Compared to other unsupervised log dataset extraction tools developed in prior work, DATAMARAN does not require the record boundaries to be known beforehand, making it much more applicable to the noisy log files that are ubiquitous in data lakes. DATAMARAN can successfully extract structured information from all datasets used in prior work, and can achieve 95% extraction accuracy on automatically collected log datasets from GitHub---a substantial 66% increase of accuracy compared to unsupervised schemes from prior work. Our user study further demonstrates that the extraction results of DATAMARAN are closer to the desired structure than competing algorithms.

Subfields: Computer Networks and Communications, Information Systems, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Management Science and Operations Research

Citations in the corpus

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, methodology): “The individual techniques are classiﬁed in Table 1. 4 Task Approach Techniques Data discovery Sharing Collaborative Analysis [9]–[11] Web [12]–[17] Collaborative and Web [18] Searching Data Lake [19]–[23] Web [24]–[34] Data augmentation Deriving Latent Semantics [35]–[37] Entity Augmentation [30], [”
Tools and Benchmarks for Automated Log Parsing (2018, methodology): “Gao et al. [54] apply an optimization algorithm to discover multi-line structures from logs.”
Data Lake Management: Challenges and Opportunities (2019, background): “The recent DATAMARAN project extracts relational data from semi-structured log ﬁles [17].”