NADEEF: a commodity data cleaning system

By Michele Dallachiesa, Amr Ebaid, Ahmed Eldawy, Ahmed K. Elmagarmid, Ihab F. Ilyas, Mourad Ouzzani, Nan Tang. ACM SIGMOD Conference, 2013

351 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/110e733d36f3232403b10f2198123ca48d838925

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Trends in Cleaning Relational Data: Consistency and Deduplication (2015, methodology): “We call the proposals that take a more holistic view of the data cleaning process Holistic cleaning approaches [26, 32, 55, 51, 48] Holistic repairing algorithms consider violations coming from different types of ICs at the same time, while suggesting updates to repair the underlying data.”
The Data Civilizer System (2017, neutral): “…DataXFormer [6] automate some aspects of cleaning and transforming data, Data Tamer [21] mainly performs schema mappings and record linkage, NADEEF [10] and BigDansing [17] provide generic programming interface for users to deﬁne their data quality rules, and DeepDive [20] extracts facts and…”
BigDansing: A System for Big Data Cleansing (2015, background): “We discuss the data cleansing semantics expressed by Big-Dansing and then give an overview of the system.”
Wisteria: Nurturing Scalable Data Cleaning Infrastructure (2015, background): “However, they are limited to specific cleaning tasks such as simple text transformations, do not support crowd-based processing at scale, and cannot incorporate user feedback to optimize the physical implementation of the data cleaning sequences.”
Interactive and Deterministic Data Cleaning (2016, background): “Rule-based approaches for data cleaning are divided between methods to discover the rules from clean data [11, 12, 17, 25, 40], and algorithms and systems to apply the rules over dirty data to automatically fix the detected errors [7,13,15,18,19,21,23,24,30,32,43,46].”
SampleClean: Fast and Reliable Analytics on Dirty Data (2015, background): “Data Cleaning: There have been many studies on various data-cleaning techniques, such as rule-based approaches [21, 18], outlier detection [26, 19], ﬁlling missing values, and duplicate detection [13, 9, 44].”
Towards Automated Data Cleaning Workflows (2019, methodology): “Outlier detection (Histograms) [22] 0.28 0.01 0.02 Outlier detection (Gaussian) [22] 0.32 0.18 0.23 Nadeef (FD) [6] 0 multiple ﬁrst-level models for error detection.”
Profiling relational data: a survey (2015, background): “Most research systems require users to supply data quality rules and dependencies, such as GDR [138], Nadeef [34], Semandaq [45] and Stream-Clean [84].”
QuERy: A Framework for Integrating Entity Resolution with Query Processing (2015, background): “The focus of QuERy is thus, complementary (though diﬀerent) to that of [23] and [9].”
KATARA: A Data Cleaning System Powered by Knowledge Bases and Crowdsourcing (2015, methodology): “For comparison with automatic data cleaning approaches, we used an equivalence-class [2] ( i.e., EQ ) based approach provided by an open-source data cleaning tool NADEEF [12], and a ML-based approach SCARE [43].”
HoloClean: Holistic Data Repairs with Probabilistic Inference (2017, methodology): “This is a benchmark dataset used in the literature [15, 18].”
Data Cleaning: Overview and Emerging Challenges (2016, background): “…FDsvaluemodiﬁcation[13] FDshypergraph[46] CFDsvaluemodiﬁcation[25] Holisticdatacleaning[22] LLUNATIC[33] Recordmatchinganddatarepairing[31] NADEEF[26] Generateoptimaltablaux[36] Uniﬁedrepair[19] Relativetrust[11] Continuousdatacleaning[73] Potter’sWheel[64] GDR[82] KATARA[23] DataTamer[69]…”
Detecting Data Errors: Where are we and what needs to be done? (2016, methodology): “In the last two decades there has been intensive research in developing data cleaning algorithms and tools [10,11,14, • Rule-based detection algorithms [1,6,15,16,25,33] that can be embedded into frameworks, such as Nadeef [8, 23], where a rule can vary from a simple “not null” constraint to multi-a”
Towards reliable interactive data cleaning: a user survey and recommendations (2016, background): “…extracttransform-load (ETL) systems [1] [2] [5] require developers to manually write data cleaning rules and execute them as long batch jobs, and constraint-driven tools allow analysts to define " data quality rules " and automatically propose corrections to maximally satisfy these rules [13].”
An Empirical Analysis of Backward Compatibility in Machine Learning Systems (2020, background): “e we show how example forgetting events relate to model incompatibility. Datacleaninganddistributionalshifts.Data cleaning is a topic of long-term interest in the data management and mining community [8, 33, 35]. Techniques addressing problems like outlier detection and data denoising are often agno”