Discovering data quality rules

By Fei Chiang, Renée J. Miller. Proceedings of the VLDB Endowment, 2008

301 citations recorded by the source corpus.

Dirty data is a serious problem for businesses leading to incorrect decision making, inefficient daily operations, and ultimately wasting both time and money. Dirty data often arises when domain constraints and business rules, meant to preserve data consistency and accuracy, are enforced incompletely or not at all in application code. In this work, we propose a new data-driven tool that can be used within an organization's data quality management process to suggest possible rules, and to identify conformant and non-conformant records. Data quality rules are known to be contextual, so we focus on the discovery of context-dependent rules. Specifically, we search for conditional functional dependencies (CFDs), that is, functional dependencies that hold only over a portion of the data. The output of our tool is a set of functional dependencies together with the context in which they hold (for example, a rule that states for CS graduate courses, the course number and term functionally determines the room and instructor). Since the input to our tool will likely be a dirty database, we also search for CFDs that almost hold. We return these rules together with the non-conformant records (as these are potentially dirty records). We present effective algorithms for discovering CFDs and dirty values in a data instance. Our discovery algorithm searches for minimal CFDs among the data values and prunes redundant candidates. No universal objective measures of data quality or data quality rules are known. Hence, to avoid returning an unnecessarily large number of CFDs and only those that are most interesting, we evaluate a set of interest metrics and present comparative results using real datasets. We also present an experimental study showing the scalability of our techniques.

Subfields: Information Systems, Management Science and Operations Research, Signal Processing

View Paper: https://www.semanticscholar.org/paper/d69b9d9ff1fd5f5232e2624b8b343bae44c949da

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Management Science and Operations Research

In Signal Processing

Citations in the corpus

Discovering Denial Constraints (2013, background): “Since DCs subsume FDs and CFDs, it is natural to ask whether we can perform reasoning the same way.”
Holistic data cleaning: Putting violations into context (2013, background): “Another subject worth of future study is how to automatically derive denial constraints from data, similarly to what has been done for other quality rules [7], [15], since experts designed constraints are not always readily available.”
Geospatial Big Data Handling Theory and Methods: A Review and Research Challenges (2015, background): “Behind this challenge is the lack of thorough knowledge of the data semantics when data that is commonly called “dirty” (e.g., Chiang & Miller 2008) is collected and combined in an ad-hoc manner.”
Interactive and Deterministic Data Cleaning (2016, background): “Rule-based approaches for data cleaning are divided between methods to discover the rules from clean data [11, 12, 17, 25, 40], and algorithms and systems to apply the rules over dirty data to automatically fix the detected errors [7,13,15,18,19,21,23,24,30,32,43,46].”
Trends in Cleaning Relational Data: Consistency and Deduplication (2015, methodology): “The second method, named CTANE, extends TANE for FD discovery and uses a level-wise search strategy, which is similar to the search strategy used in [22].”
KATARA: A Data Cleaning System Powered by Knowledge Bases and Crowdsourcing (2015, background): “Numerous studies have attempted to discover data quality rules, e.g., for CFDs [6] and for DCs [8].”
Profiling relational data: a survey (2015, methodology): “Similar interestingness functions for Cfd s are given by Chiang and Miller [24].”