Utility data annotation with Amazon Mechanical Turk

By A. Sorokin, D.A. Forsyth. 2008

714 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Evaluation & Benchmarks

In Signal Processing

Citations in the corpus

ImageNet Large Scale Visual Recognition Challenge (2014, methodology): “ILSVRC makes extensive use of Amazon Mechanical Turk to obtain accurate annotations (Sorokin and Forsyth, 2008).”
Extreme Clicking for Efficient Object Annotation (2017, background): “They report 39% efficiency gains over consensus-based approaches (which collect multiple annotations to ensure quality) [13, 60].”
CIDEr: Consensus-based image description evaluation (2014, background): “Recent advances in object recognition [15], attribute classification [23], action classification [26, 9] and crowdsourcing [40] have increased the interest in solving higher level scene understanding problems.”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, background): “3.3 Quality Control Since worker quality on AMT is highly variable (Sorokin and Forsyth 2008; Rashtchian et al. 2010), we take a combination of measures to ensure the integrity of annotations.”
Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks (2008, positive): “Sorokin and Forsyth (2008) collect data for machine vision tasks and report speed and costs similar to our findings; their summaries of worker behavior also corroborate with what we have found.”
The Language Demographics of Amazon Mechanical Turk (2014, neutral): “, 2010), computer vision (Sorokin and Forsyth, 2008; Deng et al., 2010; Rashtchian et al., 2010), speech pro-”
A data-driven approach to cleaning large face datasets (2014, background): “Even then, there is no guarantee that things will go as planned [3,4].”
See, Hear, and Read: Deep Aligned Representations (2017, methodology): “nd text by using images as a bridge (which our experiments suggest happens). To evaluate this, we also collected 1;000 text descriptions of videos (image/sound) from workers on Amazon Mechanical Turk [38], which we only use for testing the ability to transfer between sound and text. 3. Cross-Modal N”
Interestingness Prediction by Robust Learning to Rank (2014, background): “Many studies [23,40,36,30] highlight the necessity of validating the random or malicious labels/workers and give some filtering heuristics for data collection.”
SUN attribute database: Discovering, annotating, and recognizing scene attributes (2012, background): “The space of attributes is effectively infinite but the majority of possible attributes (e.g., “Was this photo taken on a Tuesday”, “Does this scene contain air?”) are not interesting.”
From QoS to QoE: A Tutorial on Video Quality Assessment (2015, background): “A more cost-effective alternative is to conduct subjective test through Internet crowdsourcing platforms, such as Amazon Mechanical Turk (MTurk) [62].”
weedNet: Dense Semantic Weed Classification Using Multispectral Images and MAV for Smart Farming (2017, background): “Therefore, it may be difficult to use coarse outsourced services [34].”
A Survey of Research on Cloud Robotics and Automation (2015, background): “11). research and many in industry, with the ROS Industrial project created to support these users [17].”
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices (2021, background): “Research has shown how crowdsourcing could be leveraged in a wide range of tasks and domains (e.g., from labeling data for ML [73, 74] to serving as a platform for experimental research [11, 47, 53]).”
Apparel Classification with Style (2012, background): “One way to achieve this is to outsource the labeling task to crowdsourcing platforms, such as Mechanical Turk [25].”
e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations (2020, methodology): “Moreover, we implemented additional quality control measures for crowdsourced annotations, such as (a) collecting three annotations for every input, (b) injecting trusted annotations into the task for verification [10], and (c) restricting to workers with at least 90% previous approval rate.”