Data Management Challenges in Production Machine Learning

By Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, Martin A. Zinkevich. 2017

243 citations recorded by the source corpus.

The tutorial discusses data-management issues that arise in the context of machine learning pipelines deployed in production. Informed by our own experience with such largescale pipelines, we focus on issues related to understanding, validating, cleaning, and enriching training data. The goal of the tutorial is to bring forth these issues, draw connections to prior work in the database literature, and outline the open research questions that are not addressed by prior art.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “plays a role in almost all aspects of machine learning [4], [5].”
Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach (2019, methodology): “More recently, there are efforts to improve machine learning accuracy [5] and data validation techniques for machine learning pipelines [10].”
Machine Learning Testing: Survey, Landscapes and Horizons (2019, unknown): “Nevertheless, data testing is challenging [220].”
Data collection and quality challenges in deep learning: a data-centric AI perspective (2021, methodology): “Schema-based validation [13,117] is widely used in practice.”
Software Engineering for Machine Learning: A Case Study (2019, background): “Since many machine learning techniques are centered around learning from large datasets, the success of ML-centric projects often heavily depends on data availability, quality and management [28].”
Automating Large-Scale Data Quality Verification (2018, background): “…involved in building complex machine learning applications in the real-world have recently been highlighted by many researchers, e.g., with respect to managing the resulting models [26], software engineering [42, 8], pipeline abstractions [2, 43, 46], and learnings from real-world systems [34, 7].”
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, background): “Multiple recent studies examine the challenges of production machine learning [6, 100, 101].”
Are Key-Foreign Key Joins Safe to Avoid when Learning High-Capacity Classifiers? (2017, background): “Furthermore, recent reports of Google’s production ML systems show that features that yield marginal benefits incur high “technical debt” that decreases code mangeability and increases costs [41, 34].”
PrIU: A Provenance-Based Approach for Incrementally Updating Regression Models (2020, background): “Closest to our perspective is the idea of influence function [29] (similar problem is also mentioned in [43]), which originates from deletion diagnostics in statistics [9].”