BoostClean: Automated Error Detection and Repair for Machine Learning

By Sanjay Krishnan, M. Franklin, Ken Goldberg, Eugene Wu. 2017

116 citations recorded by the source corpus.

Predictive models based on machine learning can be highly sensitive to data error. Training data are often combined with a variety of different sources, each susceptible to different types of inconsistencies, and new data streams during prediction time, the model may encounter previously unseen inconsistencies. An important class of such inconsistencies is domain value violations that occur when an attribute value is outside of an allowed domain. We explore automatically detecting and repairing such violations by leveraging the often available clean test labels to determine whether a given detection and repair combination will improve model accuracy. We present BoostClean which automatically selects an ensemble of error detection and repair combinations using statistical boosting. BoostClean selects this ensemble from an extensible library that is pre-populated general detection functions, including a novel detector based on the Word2Vec deep learning model, which detects errors across a diverse set of domains. Our evaluation on a collection of 12 datasets from Kaggle, the UCI repository, real-world data analyses, and production datasets that show that Boost- Clean can increase absolute prediction accuracy by up to 9% over the best non-ensembled alternatives. Our optimizations including parallelism, materialization, and indexing techniques show a 22.2x end-to-end speedup on a 16-core machine.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Val Tannen
2. Philipp Schmidt
3. Mark Harman
4. Lei Ma
5. Felix Biessmann
6. Sebastian Schelter
7. Yinjun Wu
8. Yang Liu

In Evaluation & Benchmarks

1. Mark Harman
2. Yinjun Wu
3. Yang Liu
4. Susan B. Davidson
5. Jie M. Zhang

In ML Systems & Hardware

1. Val Tannen
2. Mark Harman
3. Yinjun Wu
4. Yang Liu
5. Susan B. Davidson

Citations in the corpus

• A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, neutral): “Improve Data Data Cleaning [158], [159], [160], [161], [162], [163], [164], [165], [166] Re-labeling [119]”
• Machine Learning Testing: Survey, Landscapes and Horizons (2019, background): “12 These tables do not list datasets adopted in data cleaning evaluation, because such studies usually involve hundreds of data sets [180] 14 http://fairness.”
• Automating Large-Scale Data Quality Verification (2018, methodology): “Boost-Clean automatically selects an ensemble of error detection and repair combinations using statistical boosting [25].”
• PrIU: A Provenance-Based Approach for Incrementally Updating Regression Models (2020, background): “1007/978-1-4612-0599-9_8 [32] Sanjay Krishnan, Michael J Franklin, Ken Goldberg, and Eugene Wu.”
• “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, unknown): “and clean data include ActiveClean and BoostClean [70, 72].”