Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff
By Souvik Bhattacherjee, Amit Chavan, Silu Huang, Amol Deshpande, Aditya G. Parameswaran. 2015
86 citations recorded by the source corpus.
The relative ease of collaborative data science and analysis has led to a proliferation of many thousands or millions of versions of the same datasets in many scientific and commercial domains, acquired or constructed at various stages of data analysis across many users, and often over long periods of time. Managing, storing, and recreating these dataset versions is a non-trivial task. The fundamental challenge here is the storage-recreation trade-off: the more storage we use, the faster it is to recreate or retrieve versions, while the less storage we use, the slower it is to recreate or retrieve versions. Despite the fundamental nature of this problem, there has been a surprisingly little amount of work on it. In this paper, we study this trade-off in a principled manner: we formulate six problems under various settings, trading off these quantities in various ways, demonstrate that most of the problems are intractable, and propose a suite of inexpensive heuristics drawing from techniques in delay-constrained scheduling, and spanning tree literature, to solve these problems. We have built a prototype version management system, that aims to serve as a foundation to our DataHub system for facilitating collaborative data science. We demonstrate, via extensive experiments, that our proposed heuristics provide efficient solutions in practical dataset versioning scenarios.
Subfields: Computer Networks and Communications, Information Systems
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
In Information Systems
Citations in the corpus
- ProvDB: A System for Lifecycle Management of Collaborative Analysis Workflows (2016, background): “A version, identified by an ID, is immutable and any update to a version conceptually results in a new version with a different version ID (physical data structures are typically not immutable and the underlying DVCS uses various strategies for compact storage [4]).”
- Goods: Organizing Google's Datasets (2016, unknown): “DataHub [8, 9, 10] is a collaborative version-management system for datasets that is similar in spirit to software version control”
- Collaborative Data Analytics with DataHub (2015, unknown): “We described some of the challenges in versioning for structured data in our CIDR paper [3], developed a principled approach for trading off storage and recreation costs [4], and identified an initial query language candidate [5].”
- Data Lake Management: Challenges and Opportunities (2019, background): “One early approach DataHub provides a gitlike interface by supporting operations such as version creation, branching, merging, and viewing difference between datasets [2, 3].”