Data Lifecycle Challenges in Production Machine Learning

By Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, Martin A. Zinkevich. 2018

234 citations recorded by the source corpus.

Machine learning has become an essential tool for gleaning knowledge from data and tackling a diverse set of computationally hard tasks. However, the accuracy of a machine learned model is deeply tied to the data that it is trained on. Designing and building robust processes and tools that make it easier to analyze, validate, and transform data that is fed into large-scale machine learning systems poses data management challenges. Drawn from our experience in developing data-centric infrastructure for a production machine learning platform at Google, we summarize some of the interesting research challenges that we encountered, and survey some of the relevant literature from the data management and machine learning communities. Specifically, we explore challenges in three main areas of focus - data understanding, data validation and cleaning, and data preparation. In each of these areas, we try to explore how different constraints are imposed on the solutions depending on where in the lifecycle of a model the problems are encountered and who encounters them.

Subfields: Artificial Intelligence, Management Science and Operations Research, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Signal Processing

Citations in the corpus

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “In fact, data management plays a role in almost all aspects of machine learning [4], [5].”
The Eﬀects of Data Quality on Machine Learning Performance (2022, background): “Researchers pointed out such challenges, including data quality issues [29], data life cycle concerns [54], the connection to ML-OPs [56], and model management [64].”
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, background): “Multiple recent studies examine the challenges of production machine learning [6, 100, 101].”
Addressing "Documentation Debt" in Machine Learning: A Retrospective Datasheet for BookCorpus (2021, background): “” Although prominent research tends to be model-focused, multiple reviews [34, 37] suggest that data management creates many challenges and takes up a substantial portion of many ML workflows, including dataset generation, preprocessing, and augmentation.”
The effects of data quality on machine learning performance on tabular data (2022, unknown): “Researchers pointed out such challenges, including data quality issues [28], data life cycle concerns [55], the connection to ML-OPs [57], and model management [61].”
Challenges in Deploying Machine Learning: A Survey of Case Studies (2020, unknown): “[15].”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “cle stages is sometimes debated, for the current purposes it 1Nissenbaum’s fourth concern is about liability, which is beyond the scope of this paper. 2We differ from the “data lifecycle” analysis of [112] in that they consider the “first step of ML is to understand your data”, presuming that data a”