Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm

By A. P. Dawid, A. Skene. 1979

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Snorkel: Rapid Training Data Creation with Weak Supervision (2017, background): “Researchers have therefore considered how to estimate the accuracy of label sources without a gold standard with which to compare—a classic problem [13]—and combine these estimates into labels that can be used to train an arbitrary end model.”
Are we done with ImageNet? (2020, methodology): “In order to account for raters’ varying characteristics, we use the classic method by Dawid and Skene [38] which infers individual rater’s error rates via maximum-likelihood.”
Concrete Problems in AI Safety (2016, background): “147], one can approach unsupervised risk estimation by positing certain conditional independencies in the distribution of errors, and using this to estimate the error distribution from unlabeled data [39, 170, 121, 74]. Instead of assuming independence, another assumption is that the errors are Gaus”
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, background): “The crowdsourcing literature has studied how to increase accuracy of noisy labelers via rater modeling [11, 56] or clever construction of easier tasks based on attributes, parts, or visual similarity [55, 6, 36].”
Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks (2008, background): “Kaisser and Lowe (2008) use AMT to help build a dataset for question answering, annotating the answers to 8107 questions with the sentence containing the answer.”
Data Programming: Creating Large Training Sets, Quickly (2016, background): “Of particular relevance to our problem setting is the theoretical question of how to model the accuracy of various experts without ground truth available, classically raised in the context of crowdsourcing [9].”
DynaSent: A Dynamic Benchmark for Sentiment Analysis (2020, background): “Dawid and Skene (1979) is an early contribution to this problem leveraging Expectation Maximization (Dempster et al., 1977).”
A Survey on Truth Discovery (2015, background): “However, workers’ quality are quite diverse, which brings the core task of inferring true labels from the labeling eﬀorts of multiple workers [6,10,24,36,39,40, 43, 50, 51, 59].”
Machine Learning and Decision Support in Critical Care (2016, background): “Dawid and Skene [121] first proposed a model to probabilistically combine multiple expert annotations in an application to identify patients fit for general anaesthesia.”
Cleaning Crowdsourced Labels Using Oracles For Statistical Classification (2018, methodology): “Based on this idea, existing work treats confusion matrices and the prior as unknown parameters and adopts an EM algorithm [10] to iteratively estimate their values.”
Learning From Crowds (2010, methodology): “This algorithm is similar to the one proposed by Dawid and Skene (1979) and Smyth et al. (1995).”
Variational Inference for Crowdsourcing (2012, methodology): “One can also extend this derivation to two-coin models with independent Beta priors, yielding the EM update in Dawid and Skene [2].”
CrowdER: Crowdsourcing Entity Resolution (2012, methodology): “Instead we adopted the EM-based algorithm [9], which has been shown to be eﬀective in previous work [16,19].”
Inferring Generative Model Structure with Static Analysis (2017, background): “Estimating the accuracies of these sources without access to ground truth labels is a classic problem [13].”
Comprehensive and reliable crowd assessment algorithms (2014, background): “B) We present experimental results on synthetic data (Section IV-B) and 3 real datasets (Section IV-C) to study the performance of our confidence intervals on k-ary tasks.”
Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale (2018, background): “In response, many machine learning developers have increasingly turned to weaker methods of supervision, in which a larger volume of cheaper, noisier labels is used [6, 9, 10, 23, 24, 29, 40].”
Artificial Intelligence and Collective Intelligence (2014, methodology): “As a first example, we consider early work by Dawid and Skene (Dawid and Skene 1979).”