Aurum: A Data Discovery System

By Raul Castro Fernandez, Ziawasch Abedjan, F. Koko, Gina Yuan, Samuel Madden, M. Stonebraker. 2018

278 citations recorded by the source corpus.

Organizations face a data discovery problem when their analysts spend more time looking for relevant data than analyzing it. This problem has become commonplace in modern organizations as: i) data is stored across multiple storage systems, from databases to data lakes, to the cloud; ii) data scientists do not operate within the limits of well-defined schemas or a small number of data sources—instead, to answer complex questions they must access data spread across thousands of data sources. To address this problem, we capture relationships between datasets in an enterprise knowledge graph (EKG), which helps users to navigate among disparate sources. The contribution of this paper is AURUM, a system to build, maintain and query the EKG. To build the EKG, we introduce a Two-step process which scales to large datasets and requires only one-pass over the data, avoiding overloading the source systems. To maintain the EKG without re-reading all data every time, we introduce a resource-efficient sampling signature (RESS) method which works by only using a small sample of the data. Finally, to query the EKG, we introduce a collection of composable primitives, thus allowing users to define many different types of discovery queries. We describe our experience using AURUM in three corporate scenarios and do a performance evaluation of each component.

Subfields: Artificial Intelligence, Computer Networks and Communications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Management Science and Operations Research

Citations in the corpus

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, neutral): “AURUM [78], [79] supports data discovery queries on semantically-linked datasets.”
Seeping Semantics: Linking Datasets Using Word Embeddings for Data Discovery (2018, methodology): “We implemented SEMPROP as part of Aurum[21]1, our data discovery prototype, and evaluated its effectiveness in real scenarios.”
Data Lake Management: Challenges and Opportunities (2019, background): “On the other hand, in data-driven discovery, a user navigates a data lake (represented as a linkage graph [39, 37, 15, 14] or a hierarchical structure [26]) to ﬁnd data of her interest.”
Data collection and quality challenges in deep learning: a data-centric AI perspective (2021, methodology): “Data discovery is the problem of indexing and searching datasets that exist either in corporate data lakes [63, 157] or the Web [38].”
Data Integration: The Current Status and the Way Forward (2018, unknown): “This will require a much more elaborate catalog system, and a step in this direction is indicated in the Aurum project [5, 6], where such relationships are automatically discovered using syntactic and semantic features of the various tables and columns.”