DataHub: Collaborative Data Science & Dataset Version Management at Scale

By Anant P. Bhardwaj, Souvik Bhattacherjee, Amit Chavan, Amol Deshpande, Aaron J. Elmore, Samuel Madden, Aditya G. Parameswaran. 2014

191 citations recorded by the source corpus.

Relational databases have limited support for data collaboration, where teams collaboratively curate and analyze large datasets. Inspired by software version control systems like git, we propose (a) a dataset version control system, giving users the ability to create, branch, merge, difference and search large, divergent collections of datasets, and (b) a platform, DATAHUB, that gives users the ability to perform collaborative data analysis building on this version control system. We outline the challenges in providing dataset version control at scale.

Subfields: Information Systems, Information Systems and Management, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Information Systems and Management

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

Datasheets for datasets (2018, background): “Although data provenance has been studied extensively in the databases community [3, 6], it is rarely discussed in the machine learning community, and documenting the creation and use of datasets has received even less attention.”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Data discovery Sharing Collaborative Analysis [9], [10], [11] Web [12], [13], [14], [15], [16], [17] Collaborative and Web [18]”
Data-centric Artificial Intelligence: A Survey (2023, unknown): “Given a human-specified query (e.g., the expected attribute names), dataset discovery aims to identify the most related and useful datasets from a data lake, a repository of datasets stored in its raw formats, such as public data-sharing platforms [20] and data marketplaces.”
The role of big data analytics in Internet of Things (2017, methodology): “Cloudera introduced the Enterprise Hub [37], a Hadoop-based framework big IoT data processing and analytics be utilized as a central point managing massive amounts of IoT data enterprises.”
Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards (2021, background): “Despite existing literature on database documentation in HCI and related ﬁelds (Cheney et al., 2009; Bhardwaj et al., 2014), documentation in ML has only recently gained traction.”
YummyData: providing high-quality open life science data (2018, background): “io/), a CKANbased platform for indexing datasets (14), which offer advanced features such as version management and providing metadata in various formats.”
Ground: A Data Context Service (2017, background): “DataHub [4] is a research project that offers hosted and versioned storage of datasets, much like GitHub hosts code.”
Collaborative Data Analytics with DataHub (2015, background): “The versioning infrastructure is related to temporal databases (see [3]), however, temporal databases only allow a linear chain of versions (not conducive to collaboration).”
ProvDB: A System for Lifecycle Management of Collaborative Analysis Workflows (2016, methodology): “P ROV DB is being developed in conjunction with DataHub [3], a dataset-centric platform for enabling collaborative data analytics that supports managing a large number of datasets, their versions over time, and derived data products.”
Goods: Organizing Google's Datasets (2016, unknown): “DataHub [8, 9, 10] is a collaborative version-management system for datasets that is similar in spirit to software version control”
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, background): “Data documentation, which is a crucial aspect of facilitating collaboration, is well studied in the database and data management community [19, 23].”
How do Data Science Workers Collaborate? Roles, Workflows, and Tools (2020, methodology): “As one example, the DataHub project [8] replicates GitHublike features of version control and provenance management but for datasets.”
Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff (2015, methodology): “We have built a prototype version management system, that will serve as a foundation to DataHub [12].”
Synthetic Data for Social Good (2017, background): “Data sharing systems, including SQLShare [13] and DataHub [3], aim to facilitate collaborative data analysis, but do not incorporate privacy preserving features or purport to manage sensitive data.”