Data Cleaning: Problems and Current Approaches.

By Erhard Rahm, Hong Hai Do. 2000

1,718 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

Citations in the corpus

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “It aims to detect and correct errors and inconsistencies in the data, typically until “cleaned” data can pass consistency or validation tests (Rahm and Do, 2000).”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Common error classes are missing values in the input data, redundant entries, invalid values or broken links between entries of multiple data sets (Rahm and Do, 2000).”
AlphaClean: Automatic Generation of Data Cleaning Pipelines (2019, background): “Data cleaning is nearly as old as the relational model [13], and numerous research and commercial systems have been proposed to improve data cleaning efficiency and accuracy (see [37] for a survey).”
Survey on Automated Machine Learning (2019, background): “Common error classes are missing values in the input data, invalid values or broken links between entries of multiple data sets (Rahm and Do, 2000).”
BoostClean: Automated Error Detection and Repair for Machine Learning (2017, background): “Data Cleaning: Since the beginning of data management, several research and commercial systems have been proposed to improve data cleaning efﬁciency and accuracy (see [42] for a survey).”
HoloClean: Holistic Data Repairs with Probabilistic Inference (2017, background): “Significant efforts have been made to automate both tasks, and several surveys summarize these results [23, 31, 42].”
Evaluation of entity resolution approaches on real-world match problems (2010, background): “INTRODUCTION Entity resolution (also referred to as object matching, duplicate identification, record linkage, or reference reconciliation) is a crucial task for data integration and data cleaning [10], [18], [29].”
Overview and Importance of Data Quality for Machine Learning Tasks (2020, neutral): “Data Cleaning: Under this topic, we plan to make the audience aware of data cleaning taxonomies mentioned in the literature [15, 17] which discuss data quality problems at different granularities such as multiple tables, single table, single row, single column, single cell, etc.”
ActiveClean: Interactive Data Cleaning For Statistical Modeling (2016, background): “In the data cleaning literature, error detection and error repair are treated as two distinct problems [9,13,30].”
PrivateClean: Data Cleaning and Differential Privacy (2016, methodology): “However, raw data often requires extensive pre-processing including the extraction of named entities from textual ﬁelds [8], imputation of missing values [36], and resolution of inconsistent attributes [39].”
Data Matching (2012, methodology): “When matching schemas, a careful analysis of the names, types of content and other meta-data of the available attributes can help to identify which attributes correspond to each other [24, 224].”
A Demo of the Data Civilizer System (2017, background): “For example, work in data integration and schema mapping has focused on matching the data and meta data of di ↵ erent data sources [2], while data cleaning solutions have focused primarily on cleaning speciﬁc data sources with respect to a certain type of errors [5].”
An Empirical Analysis of Backward Compatibility in Machine Learning Systems (2020, background): “e we show how example forgetting events relate to model incompatibility. Datacleaninganddistributionalshifts.Data cleaning is a topic of long-term interest in the data management and mining community [8, 33, 35]. Techniques addressing problems like outlier detection and data denoising are often agno”
A taxonomy of privacy-preserving record linkage techniques (2013, background): “The ﬁrst step of data pre-processing (data cleaning and standardization) is crucial for quality record linkage outcomes, because most real-world data contain noisy, incomplete and inconsistent data [2,4].”
Machine Learning Testing: Survey, Landscapes and Horizons (2019, background): “These approaches are not specially designed or evaluated for ML, but they can be repurposed for ML testing [229].”
COMA - A System for Flexible Combination of Schema Matching Approaches (2002, methodology): “Approximate string matching techniques [ 10] have already been employed in other fields, such as record linkage [ 20] and data cleaning [ 19], to detect duplicate database records concerning the same real-word entity, i.e. matching at the instance level.”
Stale View Cleaning: Getting Fresh Answers from Stale Materialized Views (2015, methodology): “Since data cleaning is often expensive, we proposed cleaning a sample of data and using this sample to improve the results of aggregate queries on the full dataset.”
Eliminating Fuzzy Duplicates in Data Warehouses (2002, background): “We refer to this problem of detecting and eliminating multiple distinct records representing the same real world entity as the fuzzy duplicate elimination problem , which is sometimes also called merge/purge, dedup, record linkage problems [e,g., HS95, ME97, FS69].”