Goods: Organizing Google's Datasets

By Alon Y. Halevy, Flip Korn, Natasha Noy, Christopher Olston, Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang. 2016

213 citations recorded by the source corpus.

Enterprises increasingly rely on structured datasets to run their businesses. These datasets take a variety of forms, such as structured files, databases, spreadsheets, or even services that provide access to the data. The datasets often reside in different storage systems, may vary in their formats, may change every day. In this paper, we present GOODS, a project to rethink how we organize structured datasets at scale, in a setting where teams use diverse and often idiosyncratic ways to produce the datasets and where there is no centralized system for storing and querying them. GOODS extracts metadata ranging from salient information about each dataset (owners, timestamps, schema) to relationships among datasets, such as similarity and provenance. It then exposes this metadata through services that allow engineers to find datasets within the company, to monitor datasets, to annotate them in order to enable others to use their datasets, and to analyze relationships between them. We discuss the technical challenges that we had to overcome in order to crawl and infer the metadata for billions of datasets, to maintain the consistency of our metadata catalog at scale, and to expose the metadata to users. We believe that many of the lessons that we learned are applicable to building large-scale enterprise-level data-management systems in general.

Subfields: Computer Networks and Communications, Information Systems and Management, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems and Management

In Management Science and Operations Research

Citations in the corpus

Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “GOODs [46] is a framework that has been introduced to organize structured datasets of different formats at a scale.”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Google Data Search (GOODS) [20] is a system that”
Data Lake Management: Challenges and Opportunities (2019, methodology): “An example of a large scale metadata management system is Google Dataset Search (GOODS) that extracts and collects metadata for datasets generated and used internally by Google as they are created, used, and updated [20].”
Ground: A Data Context Service (2017, background): “LinkedIn WhereHows, FINRA Herd and Google Goods [16] are metadata services built to support the evolving data workﬂows in their respective organizations.”
Data Management Challenges in Production Machine Learning (2017, background): “Having a catalog of sources and signals can provide a ﬁrst step for discovery, and recent works have considered the problem of data cataloguing in diﬀerent contexts [10, 11] and the discovery of relationships between sources.”
The Data Civilizer System (2017, methodology): “Recently, there have been efforts to automate this process, some examples are Goods [14] and InfoGather [23].”
Seeping Semantics: Linking Datasets Using Word Embeddings for Data Discovery (2018, methodology): “Goods [29] aims at finding similar datasets by harnessing query logs, instead of creating semantic links with a mechanism such as the one presented here.”
Aurum: A Data Discovery System (2018, methodology): “For example, systems such as Goods [1], Infogather [2] and Octopus [3] do not provide a ﬂexible way to support different discovery queries than those they were designed to solve.”
Google Dataset Search: Building a search engine for datasets in an open Web ecosystem (2019, methodology): “This architecture is inspired by enterprise dataset search, such as Goods [12].”
ProvDB: A System for Lifecycle Management of Collaborative Analysis Workflows (2016, background): “GOODS [18] is a dataset management system, that passively extracts metadata information from a large number of datasets within an enterprise, and allows users to ﬁnd, monitor, and analyze datasets.”
Automating Large-Scale Data Quality Verification (2018, background): “…metadata of artifacts produced during machine learning workloads, including schemas and summary statistics of the datasets used for training and testing [47, 48, 40, 29, 28, 20, 33, 41], the discovery and organization of enterprise datasets [16], and machine learning speciﬁc san-ity checks [45].”
Navigating the Data Lake with DATAMARAN: Automatically Extracting Structure from Log Datasets (2017, background): “tems such as HDFS. These datasets are typically generated automatically as log datasets output by programs, and often number in the billions, e.g., Google has 26B datasets in their shared file system [30]. This phenomenon of accumulation of log datasets within enterprises has recently been referred ”
Managing Google's data lake: an overview of the Goods system (2016, background): “A more detailed list of challenges can be found in our previous work [5].”