An Introduction to Duplicate Detection

By Felix Naumann, Melanie Herschel. An Introduction to Duplicate Detection, 2010

371 citations recorded by the source corpus.

With the ever increasing volume of data, data quality problems abound. Multiple, yet different representations of the same real-world objects in data, duplicates, are one of the most intriguing data quality problems. The effects of such duplicates are detrimental; for instance, bank customers can obtain duplicate identities, inventory levels are monitored incorrectly, catalogs are mailed multiple times to the same household, etc. Automatically detecting duplicates is difficult: First, duplicate representations are usually not identical but slightly differ in their values. Second, in principle all pairs of records should be compared, which is infeasible for large volumes of data. This lecture examines closely the two main components to overcome these difficulties: (i) Similarity measures are used to automatically identify duplicates when comparing two records. Well-chosen similarity measures improve the effectiveness of duplicate detection. (ii) Algorithms are developed to perform on very large volumes of data in search for duplicates. Well-designed algorithms improve the efficiency of duplicate detection. Finally, we discuss methods to evaluate the success of duplicate detection. Table of Contents: Data Cleansing: Introduction and Motivation / Problem Definition / Similarity Functions / Duplicate Detection Algorithms / Evaluating Detection Success / Conclusion and Outlook / Bibliography

Subfields: Information Systems, Information Systems and Management, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/a89ad05ea07e2b5cec474a1a2c29e03f0998c5e3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Information Systems and Management

In Management Science and Operations Research

Citations in the corpus

A taxonomy of privacy-preserving record linkage techniques (2013, background): “These rules are then applied on the comparison vectors to classify candidate record pairs into matches, non-matches, or possible matches (if desired) [14,15,60].”
Detecting Data Errors: Where are we and what needs to be done? (2016, methodology): “• Record linkage and de-duplication algorithms for detecting duplicate data records [13, 26], such as the Data Tamer system [30] and its commercial descendant Tamr.”
AIDA (2011, neutral): “The emphasis has been on scalable batch processing, as needed for de-duplication [13].”
Effective Data Cleaning with Continuous Evaluation (2016, background): “A large body of work has addressed some of the well-deﬁned problems; for example, refer to the survey [11] on cleaning relational data with violations of integrity constraints, and to the multiple surveys on record linkage and de-duplication [14, 9, 8, 15, 10], and also on data proﬁling [1].”
Trends in Cleaning Relational Data: Consistency and Deduplication (2015, background): “The topic has been extensively covered in many surveys [81, 40, 39, 90, 58, 70]: some aim at providing an extensive overview of all the steps involved in data deduplication [40, 58, 70], some focus on the design of similarity metrics [90, 81], some discuss the efficiency aspect of data deduplication”
A Comprehensive Study of the Past, Present, and Future of Data Deduplication (2016, neutral): “In these fields, duplicate detection is commonly used to identify records that represent the same entity [33], [34].”
Deep Learning for Entity Matching: A Design Space Exploration (2018, background): “Related EMWork: EM has received much attention [9, 19, 24, 55, 63].”
Data Integration: The Current Status and the Way Forward (2018, background): “This step is crucial in enabling comparing records across sources; (e) Deduplication and record linkage tools [4, 7, 10] are used to cluster records.”
HoloClean: Holistic Data Repairs with Probabilistic Inference (2017, methodology): “For error detection, many methods rely on violations of integrity constraints [11, 14] or duplicate [27, 37, 40] and outlier detection [19, 29] methods to identify errors.”
Distributed Representations of Tuples for Entity Resolution (2017, background): “ER is typically treated as a binary classification problem [22, 20, 42, 7].”