ProvDB: A System for Lifecycle Management of Collaborative Analysis Workflows

By Hui Miao, Amit Chavan, Amol Deshpande. 2016

16 citations recorded by the source corpus.

As data-driven methods are becoming pervasive in a wide variety of disciplines, there is an urgent need to develop scalable and sustainable tools to simplify the process of data science, to make it easier to keep track of the analyses being performed and datasets being generated, and to enable introspection of the workflows. In this paper, we describe our vision of a unified provenance and metadata management system to support lifecycle management of complex collaborative data science workflows. We argue that a large amount of information about the analysis processes and data artifacts can, and should be, captured in a semi-passive manner; and we show that querying and analyzing this information can not only simplify bookkeeping and debugging tasks for data analysts but can also enable a rich new set of capabilities like identifying flaws in the data science process itself. It can also significantly reduce the time spent in fixing post-deployment problems through automated analysis and monitoring. We have implemented an initial prototype of our system, called ProvDB, on top of git (a version control system) and Neo4j (a graph database), and we describe its key features and capabilities.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Data Lifecycle Challenges in Production Machine Learning (2018, background): “ProvDB [49] provides a uniﬁed provenance and metadata management system to support lifecycles of complex collaborative data science workﬂows.”
Ground: A Data Context Service (2017, background): “A very recent technical report on ProvDB [24] echoes some of Ground’s vision of coupling versioning and lineage, and proposes an architectural shim on top of a versioned store like git or DataHub.”