Comprehensive and reliable crowd assessment algorithms

By Manas R. Joglekar, Hector Garcia-Molina, Aditya G. Parameswaran. 2014

53 citations recorded by the source corpus.

Evaluating workers is a critical aspect of any crowdsourcing system. In this paper, we devise techniques for evaluating workers by finding confidence intervals on their error rates. Unlike prior work, we focus on “conciseness”-that is, giving as tight a confidence interval as possible. Conciseness is of utmost importance because it allows us to be sure that we have the best guarantee possible on worker error rate. Also unlike prior work, we provide techniques that work under very general scenarios, such as when not all workers have attempted every task (a fairly common scenario in practice), when tasks have non-boolean responses, and when workers have different biases for positive and negative tasks. We demonstrate conciseness as well as accuracy of our confidence intervals by testing them on a variety of conditions and multiple real-world datasets.

Subfields: Artificial Intelligence, Computer Science Applications, Transportation, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Transportation

In Evaluation & Benchmarks

Citations in the corpus

Snorkel: Rapid Training Data Creation with Weak Supervision (2017, background): “Much of this work has focused on crowdsourcing, in which workers have unknown accuracy [11, 23, 53].”
Data Programming: Creating Large Training Sets, Quickly (2016, background): “More recent results provide formal guarantees even in the absence of labeled data using various approaches [3,8,14,15,22,31].”
SLiMFast: Guaranteed Results for Data Fusion and Source Reliability (2015, background): “The only work prior to ours that comes with theoretical guarantees on the error rate of accuracy estimates is from the crowdsourcing community [8, 17, 20, 18] where data sources correspond to human workers.”
Snuba: Automating Weak Supervision to Label Training Data (2018, background): “Combining labels from multiple sources like heuristics is well-studied problem [11], especially in the context of crowdsourcing [9, 23, 61].”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, background): “These papers consider increasingly expressive models for this estimation problem, including worker bias [100], difficulty of tasks and worker expertise [193, 157, 106, 108], adversarial behavior [156], and online evaluation of workers [191, 128, 155].”
Socratic Learning: Augmenting Generative Models to Incorporate Latent Subsets in Training Data (2016, background): “Estimating the accuracies of these sources without access to ground truth labels is a classic problem [9], especially in the ﬁeld of crowdsourcing [10, 7, 14, 33].”
Learning the Structure of Generative Models without Labeled Data (2017, methodology): “Many methods for crowdsourcing estimate the accuracy of workers without hand-labeled data (e.g., Dalvi et al., 2013; Joglekar et al., 2015; Zhang et al., 2016).”
Metrology for AI: From Benchmarks to Instruments (2019, background): “To ensure quality of the resulted crowdsourced data, various mechanisms have been developed to measure the quality of the raters; (Joglekar, Garcia-Molina, and Parameswaran 2015) provide concise confidence intervals on raters error rates, (Eickhoff 2018) focus on cognitive biases in crowdsourcing,…”
Inferring Generative Model Structure with Static Analysis (2017, background): “Methods such as crowdsourcing [12, 17, 40], boosting[3, 33], co-training [6], and learning from noisy labels are some of the popular approaches that can combine various sources of weak supervision to assign noisy labels to data.”
Challenges in Data Crowdsourcing (2016, background): “While there has been some work in taking individual worker biases into account [25], [26], [61], many papers in crowdsourcing algorithms assume a uniform error rate for all workers, thereby ignoring all worker-specific bias.”
Snorkel: rapid training data creation with weak supervision (2019, background): “Much of this work has focused on crowdsourcing, in which workers have unknown accuracy [11,25,66].”