Error Rate Analysis of Labeling by Crowdsourcing

By Hongwei Li. 2013

53 citations recorded by the source corpus.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Christopher R'e
2. Serge J. Belongie
3. Pietro Perona
4. Grant Van Horn
5. Alexander J. Ratner
6. Steve Branson
7. Paroma Varma
8. Eric P. Xing

Citations in the corpus

• Snorkel: Rapid Training Data Creation with Weak Supervision (2017, methodology): “grows, known theory confirms that the majority vote will eventually be optimal [27].”
• Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling (2020, methodology): “…some assumptions on the accuracy and error probabilities of labeling functions, similar to the Homogenous Dawid-Skene model (Dawid & Skene, 1979; Li et al., 2013) in crowd sourcing, where label source accuracy is the same across classes and errors are evenly divided with probability mass…”
• Snuba: Automating Weak Supervision to Label Training Data (2018, background): “With a large number of heuristics, each datapoint receives enough labels that learning accuracies has little effect on the assigned labels [29].”
• Snorkel: rapid training data creation with weak supervision (2019, methodology): “grows, known theory confirms that the majority vote will eventually be optimal [27].”
• Lean Multiclass Crowdsourcing (2018, methodology): “Alternate inference algorithms for the DS model are based on spectral methods [7, 4, 12, 13, 14, 40], belief propagation [20, 23], expectation maximization [20, 40], maximum entropy [41, 42], weighted majority voting [19, 17] and max-margin [32].”