Data Cleaning: Overview and Emerging Challenges

By Xu Chu, Ihab F. Ilyas, Sanjay Krishnan, Jiannan Wang. 2016

559 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Information Systems, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Information Systems

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “Data cleaning and feature engineering are ignored completely even though those two steps have a significant impact on the final performance of an ML pipeline (Chu et al., 2016).”
AlphaClean: Automatic Generation of Data Cleaning Pipelines (2019, methodology): “Many popular data cleaning libraries actually “speak the same language”, where all of their repairs can be cast as cell-replacements operations [12, 31, 41].”
Survey on Automated Machine Learning (2019, background): “hm. Data cleaning and feature engineering are completely ignored even though those two steps have a signicant impact on the nal 40 Survey on Automated Machine Learning performance of an ML pipeline (Chu et al., 2016). The only possibility to capture and evaluate all aspects of AutoML algorithms is u”
BoostClean: Automated Error Detection and Repair for Machine Learning (2017, background): “The ability for a data cleaning system to accurately identify data errors relies on the availability of a set of high-quality error detection rules [13].”
A Data Quality Metric (DQM): How to Estimate the Number of Undetected Errors in Data Sets (2016, background): “Analysts report that data cleaning is often the most time-consuming and costly step in the analysis process [1, 8].”
Trustworthy AI: From Principles to Practices (2021, methodology): “Data Preprocessing Anomaly Detection [70, 81, 257, 316] Data Anonymozation [14, 37, 232, 232] Differential Privacy [107, 110, 177]”
Cleaning Crowdsourced Labels Using Oracles For Statistical Classification (2018, background): “Algorithmic data cleaning approaches have been improving in quality, but still far from perfect [7].”
Automating Large-Scale Data Quality Verification (2018, background): “Data cleaning has been an active research area for decades, see recent surveys [1, 10, 38] for an overview.”
Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls (2022, background): “Earlier work goes back to data cleaning (Rahm and Do, 2000; Chapelle, 2004; Chu et al., 2016), surrogate losses (Natarajan et al., 2013) and probabilistic methods (Rebbapragada and Brodley, 2007).”
Towards reliable interactive data cleaning: a user survey and recommendations (2016, background): “Over the last two decades, data cleaning has been a key area of database research (see surveys [14], [34], and tutorial [11]).”
PrIU: A Provenance-Based Approach for Incrementally Updating Regression Models (2020, background): “tal techniques to efficiently update regression models arises in several contexts, for example data cleaning and interpretability. Data cleaning has been extensively studied by the database community [8, 12, 17, 46], and is typically an iterative and interactive process, allowing data analysts to al”