Embracing Error to Enable Rapid Crowdsourcing

By Ranjay Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A. Shamma, Li Fei-Fei, Michael S. Bernstein. 2016

93 citations recorded by the source corpus.

Microtask crowdsourcing has enabled dataset advances in social science and machine learning, but existing crowdsourcing schemes are too expensive to scale up with the expanding volume of data. To scale and widen the applicability of crowdsourcing, we present a technique that produces extremely rapid judgments for binary and categorical labels. Rather than punishing all errors, which causes workers to proceed slowly and deliberately, our technique speeds up workers' judgments to the point where errors are acceptable and even expected. We demonstrate that it is possible to rectify these errors by randomizing task order and modeling response latency. We evaluate our technique on a breadth of common labeling tasks such as image verification, word similarity, sentiment analysis and topic classification. Where prior work typically achieves a 0.25x to 1x speedup over fixed majority vote, our approach often achieves an order of magnitude (10x) speedup.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

VirTex: Learning Visual Representations from Textual Annotations (2020, neutral): “To collect classification labels, typically human experts first build an ontology of categories [3, 4, 30, 31], then complex crowdsourcing pipelines are used to elicit labels from non-expert users [32, 33].”
Dense-Captioning Events in Videos (2017, background): “wage rate [41]. 7.4. Annotation details Following research from previous work that show that crowd workers are able to perform at the same quality of work when allowed to video media at a faster rate [24], we show all videos to workers at 2X the speed, i.e. the videos are shown at twice the frame ra”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “Crowdsourced datasets are often subject to noise attributed to incorrect labeled annotations (Sheng et al., 2008; Krishna et al., 2016; Ekambaram et al., 2017), which may lead to training models that are not representative of the task at hand.”
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, methodology): “An image evaluation begins with a 3-2-1 countdown clock, each number displaying for 500ms [30].”
Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee (2019, background): “Therefore preventing over-fitting is crucial for robust performance since mislabeled data are ubiquitous in very large datasets (Krishna et al., 2016).”
Human Uncertainty Makes Classification More Robust (2019, methodology): “Our approach proposes utilizing these human disagreements to improve the accuracy and robustness of a model, complementing existing work aimed at leveraging “errors” in human labeling [27].”
Efficient Object Annotation via Speaking and Pointing (2019, methodology): “Let us take the ILSVRC dataset as an example: getting labels for the 200 object classes in its vocabulary would take close to 6min per image (Krishna et al. 2016), despite each image containing only 1.6 classes on average.”
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices (2021, background): “allotted workers to spend on the task naturally limits the associated cost of the experiment, besides studies in cost-effective crowdsourcing shed light on the impact of time on worker behavior and performance [41, 42].”
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Accuracy (2016, background): “Although crowdsourcing has shown to have the potential to be intrinsically motivating [54], the microtask paradigm found in the majority of crowdsourcing tasks favors a structure that is efficient [15,26] for workers rather than being interesting for them [7].”
Y Qualification Main task Y Return N N Y User qualified ? N Feedback Pass ? Pass ? Annotator training Annotating images (2017, background): “5s in [33]) indicating that most of the time is spent on the visual search to find the object and not on clicking on it.”
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets (2017, background): “…crowdsourcing has been employed for a variety of purposes including minimizing reaction time in crowd-powered user interfaces [7, 6, 9, 34, 36], increasing the speed of data collection [35, 31], and improving data quality by incorporating expert interventions to guide novice workers [11, 20, 28].”