Quantitative Data Cleaning for Large Databases

By Joseph M Hellerstein. 2008

271 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Using different representations of expert knowledge, like integrity constraints or first order logic, low quality data can be detected and corrected automatically (Raman & Hellerstein, 2001; Hellerstein, 2008; Chu et al., 2015, 2016).”
Survey on Automated Machine Learning (2019, background): “Using different representations of expert knowledge, like integrity constraints or first order logic, low quality data can be automatically detected and corrected (Raman and Hellerstein, 2001; Hellerstein, 2008; Chu et al., 2015, 2016).”
BoostClean: Automated Error Detection and Repair for Machine Learning (2017, background): “There is a well-established line of literature on statistical anomaly detection [23], and for the most part, these techniques are generic and dataset independent (up-to hyperparameters).”
HoloClean: Holistic Data Repairs with Probabilistic Inference (2017, methodology): “For error detection, many methods rely on violations of integrity constraints [11, 14] or duplicate [27, 37, 40] and outlier detection [19, 29] methods to identify errors.”
Wrangler: interactive visual specification of data transformation scripts (2011, methodology): “These techniques include methods for detecting erroneous values [10, 11], information extraction [1, 25], entity resolution [6], type inference [7], and schema matching [9, 21].”
Combining Quantitative and Logical Data Cleaning (2015, background): “In quantitative data cleaning, there has been considerable work on outlier detection for quantitative data (both univariate and multivariate or relational data) [15].”
Detecting Data Errors: Where are we and what needs to be done? (2016, background): “[17] follow on the same line when considering numerical data and surveying quantitative data cleaning approaches [17].”
A sample-and-clean framework for fast and accurate query processing on dirty data (2014, background): “Data Cleaning: There have been many studies on various data-cleaning techniques, such as rule-based approaches [17, 23], outlier detection [18,31], filling missing values [48,52], and duplicate detection [9,12].”
Automating Large-Scale Data Quality Verification (2018, background): “The metric Uniqueness refers to the unique value ratio [19] in a particular column:”
MacroBase: Analytic Monitoring for the Internet of Things (2016, unknown): “These techniques—and related data cleaning techniques [24,31]—are powerful and can handle non-”
MacroBase: Prioritizing Attention in Fast Data (2016, methodology): “The literature contains several recent explanation techniques leveraging decision-tree [23] and Apriori-like [39, 80] pruning, grid search [68, 82], data cubing [69], Bayesian statistics [78], visualization [17,62], causal reasoning [81], and several others [31, 43, 51, 58].”
Data Cleaning: Overview and Emerging Challenges (2016, background): “Quantitative data cleaning techniques have been extensively covered in multiple surveys [2, 65, 40] and tutorials [48, 17], but there have been fewer surveys of qualitative data cleaning [44].”
Trends in Cleaning Relational Data: Consistency and Deduplication (2015, methodology): “Rahm and Do [93] give a classification of different types of errors that can happen in an Extract-Transform-Load (ETL) process, and survey the tools available for cleaning data in an ETL process; some focus on the effect of incompleteness data on query answering [61], and the use of a Chase procedur”
Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (2023, background): “External Knowledge: 1) databases contain values of noisy data types [14, 23, 19, 31].”
Working capital level influence on SME profitability (2016, unknown): “The method applied in this study was to winsorise the data with outliers at the 5% and 95% levels by replacing the extreme observations with the nearest non-outlier observation (Hellerstein, 2008).”
SampleClean: Fast and Reliable Analytics on Dirty Data (2015, background): “Data Cleaning: There have been many studies on various data-cleaning techniques, such as rule-based approaches [21, 18], outlier detection [26, 19], ﬁlling missing values, and duplicate detection [13, 9, 44].”
Profiler: integrated statistical analysis and visualization for data quality assessment (2012, methodology): “The database and statistics literature includes many taxonomies of anomalous data [5, 7, 10, 19, 24, 33].”
Airavat: Security and Privacy for MapReduce (2010, background): “Differential privacy guarantees are somewhat similar to robust or secure statistical estimation, which provides statistical computations with low sensitivity to any single input ( e.g. , see [21, 23, 24]).”