BigDansing: A System for Big Data Cleansing

By Zuhair Khayyat, Ihab F. Ilyas, Alekh Jindal, Samuel Madden, Mourad Ouzzani, Paolo Papotti, Jorge-Arnulfo Quiané-Ruiz, Nan Tang, Si Yin. 2015

180 citations recorded by the source corpus.

Data cleansing approaches have usually focused on detecting and fixing errors with little attention to scaling to big datasets. This presents a serious impediment since data cleansing often involves costly computations such as enumerating pairs of tuples, handling inequality joins, and dealing with user-defined functions. In this paper, we present BigDansing, a Big Data Cleansing system to tackle efficiency, scalability, and ease-of-use issues in data cleansing. The system can run on top of most common general purpose data processing platforms, ranging from DBMSs to MapReduce-like frameworks. A user-friendly programming interface allows users to express data quality rules both declaratively and procedurally, with no requirement of being aware of the underlying distributed platform. BigDansing takes these rules into a series of transformations that enable distributed computations and several optimizations, such as shared scans and specialized joins operators. Experimental results on both synthetic and real datasets show that BigDansing outperforms existing baseline systems up to more than two orders of magnitude without sacrificing the quality provided by the repair algorithms.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

In ML Systems & Hardware

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, neutral): “, (Krishnan et al., 2015; Khayyaty et al., 2015; Krishnan et al., 2016; Eduardo and Sutton, 2016; Rekatsinas et al., 2017).”
AlphaClean: Automatic Generation of Data Cleaning Pipelines (2019, background): “cleaning [4, 23, 28, 30, 41, 45] has revealed human-time — Machine Learning in Data Cleaning: Machine learning Application-Aware Cleaning: Semantics about the down-stream Generating Cleaning Programs: A composable data”
A Survey on Cleaning Dirty Data Using Machine Learning Paradigm for Big Data Analytics (2018, methodology): “Existing procedures involve jamming data for identical data detection [29], [30], identification and linkage for data cleaning [30], clean data using sampling [31], and distributed data cleaning [32].”
BoostClean: Automated Error Detection and Repair for Machine Learning (2017, background): “Over the past few years, there have been several signiﬁcant data cleaning advances in scalability [7,28,49].”
Interactive and Deterministic Data Cleaning (2016, background): “Rule-based approaches for data cleaning are divided between methods to discover the rules from clean data [11, 12, 17, 25, 40], and algorithms and systems to apply the rules over dirty data to automatically fix the detected errors [7,13,15,18,19,21,23,24,30,32,43,46].”
Overview of data quality challenges in the context of Big Data (2015, background): “‘BigDansing’ have been benchmarked with other systems which could support some level of data cleansing routines such as Spark SQL, Shark, NADEEF and PostGreSQL.”
Trends in Cleaning Relational Data: Consistency and Deduplication (2015, methodology): “BigDansing [76] is a distributed data cleaning system that runs on top of common data processing platforms, such as DBMS, or MapReduce frameworks.”
Data Cleaning: Overview and Emerging Challenges (2016, methodology): “Current techniques include blocking for duplicate detection [5], sampling for data cleaning [75], and distributed data cleaning [47, 45, 20].”
Detecting Data Errors: Where are we and what needs to be done? (2016, methodology): “In the last two decades there has been intensive research in developing data cleaning algorithms and tools [10,11,14, • Rule-based detection algorithms [1,6,15,16,25,33] that can be embedded into frameworks, such as Nadeef [8, 23], where a rule can vary from a simple “not null” constraint to multi-a”
Towards reliable interactive data cleaning: a user survey and recommendations (2016, background): “…become easier to acquire and store ever larger datasets, the challenges associated with large-scale data cleaning, wherein issues caused by incorrect, missing, and duplicate data are identified and repaired, has become a subject of intense interest in the academic community e.g., [12] [19] [26].”