Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks
By Rion Snow, Brendan T. O'Connor, Dan Jurafsky, Andrew Y. Ng. 2008
2,515 citations recorded by the source corpus.
Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement between Mechanical Turk non-expert annotations and existing gold standard labels provided by expert labelers. For the task of affect recognition, we also show that using non-expert labels for training machine learning algorithms can be as effective as using gold standard annotations from experts. We propose a technique for bias correction that significantly improves annotation quality on two tasks. We conclude that many large labeling tasks can be effectively designed and carried out in this method at a fraction of the usual expense.
Subfields: Artificial Intelligence, Computer Science Applications, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Yejin Choi
- Noah A. Smith
- Yoshua Bengio
- Christopher D. Manning
In Computer Science Applications
- Michael S. Bernstein
- Adam Marcus
- Aditya G. Parameswaran
- Lydia B. Chilton
- Kevin Crowston
- Lilly C. Irani
- Pietro Perona
- Siddharth Suri
In Evaluation & Benchmarks
- Mohit Bansal
- Douwe Kiela
- Samuel R. Bowman
- Yejin Choi
- Luke S. Zettlemoyer
- Jianfeng Gao
- Christopher D. Manning
- Robin Jia
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “In particular, MTurk has a long history in NLP with early claims stating that a small number of crowdworkers can replace a single expert rater (Snow et al., 2008).”
- Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020, background): “, 1993), crowd-sourcing by non-experts (Snow et al., 2008), distant supervision (Mintz et al.”
- A SICK cure for the evaluation of compositional distributional semantic models (2014, background): “Snow et al. (2008) show high agreement between non-expert annotations of the RTE -1 dataset and existing gold standard labels assigned by expert annotators.”
- Dynabench: Rethinking Benchmarking in NLP (2021, background): “scale NLP dataset creation (Snow et al., 2008; Munro et al., 2010), we ultimately want NLP systems to handle “natural” data (Kwiatkowski et al.”
- TreeTalk: Composition and Compression of Trees for Image Descriptions (2014, methodology): “To learn the syntactic patterns for caption generalization, we collect a small set of example compressed captions (380 in total) using Amazon Mechanical Turk (AMT) (Snow et al., 2008).”
- Distant supervision for relation extraction without labeled data (2009, methodology): “Human evaluation was performed by evaluators on Amazon’s Mechanical Turk service, shown to be effective for natural language annotation in Snow et al. (2008).”
- Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines (2014, background): “This is the main phase of each crowdsourcing project, which on smaller paid-for crowdsourcing projects can be completed sometimes in a matter of minutes (Snow et al., 2008), or run for many months or years, when games or volunteers are used for collecting large datasets.”
- Inherent Disagreements in Human Textual Inferences (2019, background): “That is, the prevailing assumption across annotationmethods is that there is a single ‘‘true’’ inference about h given p that we should train models to predict, and that this label can be approximated by aggregating multiple (possibly noisy) human ratings as is typical in many other labelling tasks ”
- Interpretation of Natural Language Rules in Conversational Machine Reading (2018, background): “We also restrict participation to workers with > 97% approval rate, > 1000 previously completed HITs and located in the UK, US or Canada.”
- Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces (2018, methodology): “the data generation pipeline. 5.1.1 Crowdsourcing Crowdsourcing tasks were originally submitted to Amazon Mechanical Turk9, a widely used platform in non-expert annotations for natural language tasks [51]. While a sufficient number of Englishspeaking contributors can be reached easily, other language”
- On Releasing Annotator-Level Labels and Information in Datasets (2021, methodology): “Obtaining multiple annotator judgements on the same data instances is a common practice in NLP in order to improve the quality of final labels (Snow et al., 2008; Nowak and Rüger, 2010).”
- MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, background): “AMT workers sometimes submit inaccurate, random, or even adversarial decisions [35].”
- DynaSent: A Dynamic Benchmark for Sentiment Analysis (2020, methodology): “Sentiment analysis is an early success story for NLP, in both a technical and an industrial sense.”
- The Language Demographics of Amazon Mechanical Turk (2014, background): “, 2013), and natural language processing (Snow et al., 2008; CallisonBurch and Dredze, 2010; Laws et al., 2011).”
- MultiLing 2015: Multilingual Summarization of Single and Multi-Documents, On-line Fora, and Call-center Conversations (2015, methodology): “1http://www.sensei-conversation.eu/ Submissions were evaluated via crowdsourcing on Crowd Flowerwhich is a commonly used method for evaluating HLT systems (Snow et al., 2008; Callison-Burch, 2009).”
- TurKit: human computation algorithms on mechanical turk (2010, background): “A variety of platforms for human computation have been developed and studied: Games with a Purpose [1], Wikipedia [5] [12], and MTurk [10] [11] [16] [19] [20].”