crowdsourcing : rating annotators and obtaining cost-effective labels

By Peter Welinder, Pietro Perona. 2010

363 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Evaluation & Benchmarks

Citations in the corpus

Describing Textures in the Wild (2013, methodology): “Beyond simple voting, we found that the method of [38] can effectively remove or down-weigh bad annotators improving agreement.”
Towards a Visual Turing Challenge (2014, background): “Crowdsourci ng in combination of machine learning approaches have served us well to generate curated datasets with a unique ground truth at scale [33, 34].”
Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions (2019, background): “[46] deals with a different approach to attend to the diversity of human annotators through a multi-dimensional measure of the annotators’ ability.”
Part-dependent Label Noise: Towards Instance-dependent Label Noise (2020, background): “The main reason is that it is expensive and sometimes even infeasible to accurately label large-scale datasets [23]; while it is relatively easy to obtain cheap but noisy datasets [77, 62, 65, 71, 19].”
Learning From Noisy Singly-labeled Data (2017, background): “Another line of work employs active learning, iteratively filtering out examples for which aggregated labels have high confidence and collect additional labels for the remaining examples (Whitehill et al., 2009; Welinder and Perona, 2010; Khetan and Oh, 2016).”
Crowdsourcing in Computer Vision (2016, neutral): “The method of [Welinder and Perona, 2010] is applicable generally to binary, multi-valued, and even continu-”
A Survey on Security, Privacy, and Trust in Mobile Crowdsourcing (2018, background): “Many MCS applications have been developed [1-31] and are used for environment monitoring [2, 4], infrastructure monitoring quality of-experience analysis [8, 9], surface perception [5], and public safety [7].”
Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data (2016, methodology): “Welinder and Perona (2010) develop a classiﬁer that integrates data difﬁculty and worker characteristics, while Welinder et al. (2010) develop a unifying model of the characteristics of both data and workers, such as competence, expertise, and bias.”
Crowdsourcing Annotations for Visual Object Detection (2012, background): “The coverage verification task requests a third worker to check whether there are still instances of raccoon not covered by a bounding box.”
Crowdsourced Data Management: A Survey (2016, background): “E XISTING algorithms cannot effectively address computerhard tasks such as entity resolution [116], [118], [121], [122], sentiment analysis [74], [82], [139], and image recognition [104], [124], [129], which can benefit from the use of human cognitive ability.”
Annotator rationales for visual recognition (2011, background): “A substantial shift in the scale of image datasets annotated today is in part due to the emergence of online services like Mechanical Turk; one can post jobs to get a large number of labels more quickly [4, 21], and consider ways to manage quality control during collection [8, 21, 27].”
The Multidimensional Wisdom of Crowds (2010, methodology): “Other approaches have been proposed for non-binary annotations [9, 6, 12].”
CrowdScreen: algorithms for filtering data with humans (2012, background): “Similar work has considered a pool of workers where the most “informed” worker is asked for a label on the fly on the most uncertain item [15, 16], and the accuracy of the worker is learned as labels are obtained.”
Why rankings of biomedical image analysis competitions should be interpreted with care (2018, background): “[Welinder and Perona 2010, Bartlett and Frost 2008, Van den Heuvel and Trip 2002, Walter et al. 1998] How to best combine multiple annotations?”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, background): “These papers consider increasingly expressive models for this estimation problem, including worker bias [100], difficulty of tasks and worker expertise [193, 157, 106, 108], adversarial behavior [156], and online evaluation of workers [191, 128, 155].”
Comprehensive and reliable crowd assessment algorithms (2014, background): “B) We present experimental results on synthetic data (Section IV-B) and 3 real datasets (Section IV-C) to study the performance of our confidence intervals on k-ary tasks.”
Machine Learning and Decision Support in Critical Care (2016, background): “Welinder and Perona [125] proposed a similar model in a Bayesian framework, again estimating the precision (or inverse variance) associated with each annotator’s labels.”
Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection (2015, background): “Although these workers will usually be non-experts, for many tasks it has been shown that repeated labeling of examples by multiple non-expert workers can produce high quality labels [30, 37, 14].”
Large-Scale Live Active Learning: Training Object Detectors with Crawled Data and Crowds (2011, background): “Even with a human in the loop, annotation precision varies when using Web interfaces and crowds, and so some research explores ways to automatically provide quality assurance [13, 14].”