CrowdER: Crowdsourcing Entity Resolution

By Jiannan Wang, Tim Kraska, M. Franklin, Jianhua Feng. 2012

611 citations recorded by the source corpus.

Entity resolution is central to data integration and data cleaning. Algorithmic approaches have been improving in quality, but remain far from perfect. Crowdsourcing platforms offer a more accurate but expensive (and slow) way to bring human insight into the process. Previous work has proposed batching verification tasks for presentation to human workers but even with batching, a human-only approach is infeasible for data sets of even moderate size, due to the large numbers of matches to be tested. Instead, we propose a hybrid human-machine approach in which machines are used to do an initial, coarse pass over all the data, and people are used to verify only the most likely matching pairs. We show that for such a hybrid system, generating the minimum number of verification tasks of a given size is NP-Hard, but we develop a novel two-tiered heuristic approach for creating batched tasks. We describe this method, and present the results of extensive experiments on real data sets using a popular crowdsourcing platform. The experiments show that our hybrid approach achieves both good efficiency and high accuracy compared to machine-only or human-only alternatives.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

Citations in the corpus

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, neutral): “CrowdER [118] provides two types of interfaces to compare records: pair-based and cluster-based.”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, methodology): “A good body of work has developed around entity resolution and clustering [189, 86], where the focus is also on pairwise comparisons.”
Mobile Crowd Sensing and Computing (2015, methodology): “Similarly, to obtain the minimum number of veriﬁcation tasks, CrowdER [Wang et al. 2012] developed a two-tiered hybrid human–machine approach for creating batched tasks, where machines are used to do an initial, coarse pass over all the data, and people are used to verify only the most likely matchi”
Data Cleaning: Overview and Emerging Challenges (2016, background): “FDsvaluemodiﬁcation[13] Holisticdatacleaning[22] CrowdER[74] Corleone[35] CausalityAnalysis[58] Scorpion[79] DBRx[16] Table 1: A sample of qualitative error detection techniques.”
Big data challenge: a data management perspective (2013, background): “A recent work [17] tries to make use of crowd in data integration tasks.”
Deep Learning for Entity Matching: A Design Space Exploration (2018, methodology): “Finally, a different line of work develops methods to leverage human experts [26, 73, 81] to guide EM and help refine the learned matching functions.”
Trends in Cleaning Relational Data: Consistency and Deduplication (2015, methodology): “In contrast, deciding whether two records are duplicates usually requires fuzzy matching, for which humans sometimes can achieve better accuracy [114, 59, 116, 112].”
Crowdsourced Data Management: A Survey (2016, background): “EM-Based methods [15], [41], [58], [70], [79], [100], [104], [112], [118], [127], [139].”
Query-Oriented Data Cleaning with Oracles (2015, methodology): “It is a powerful tool that has been employed for database cleaning tasks such as entity/conﬂict resolution [61, 65], duplicate detection [6, 11], schema matching [67, 35, 43], and ﬁlling up missing data [50, 49, 27].”
A Data Quality Metric (DQM): How to Estimate the Number of Undetected Errors in Data Sets (2016, methodology): “To illustrate such limitations of this naïve approach, we ran an initial experiments using the restaurant data set (also used in [32, 14]).”
Online mobile Micro-Task Allocation in spatial crowdsourcing (2016, methodology): “In recent years, a wide spectrum of fundamental datadriven operations for general crowdsourcing platforms are well studied, such as filtering[13], entity resolution[14], data cleaning[15], topic discovery[16], and a couple of crowdpowered database systems have been successfully developed, such as Cr”
Can Foundation Models Wrangle Your Data? (2022, background): “EM has been extensively studied over the past decade (see [64] for a survey) and methods broadly fall into three categories: rulebased, crowd-based [31, 81] and ML/DL-based [41, 62].”
Challenges in Data Crowdsourcing (2016, methodology): “(Of course, as research in this area [13], [23], [32], [53], [57] suggests, there are much better ways to solve these problems, avoiding many of the unnecessary comparisons, while coping with human errors efficiently.)”
Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions (2015, background): “Another related problem is record linkage [34], [35], [36], [37], [38], [39], [40], [41] (also called duplicate detection, entity matching, and reference reconciliation) in the database community.”
Truth Inference in Crowdsourcing: Is the Problem Solved? (2017, methodology): “D Product [50] 8,315 8,315 24,945 3 176”
Comprehensive and reliable crowd assessment algorithms (2014, background): “B) We present experimental results on synthetic data (Section IV-B) and 3 real datasets (Section IV-C) to study the performance of our confidence intervals on k-ary tasks.”
Corleone: hands-off crowdsourcing for entity matching (2014, background): “Current works use the crowd to verify predicted matches [30, 31, 6], finds the best questions to ask the crowd [33], and finds the best UI to pose such questions [34].”
Foundations of Crowd Data Sourcing (2015, neutral): “, [9, 11, 15, 23, 24, 27, 29]).”