Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk

By Chris Callison-Burch. 2009

530 citations recorded by the source corpus.

Manual evaluation of translation quality is generally thought to be excessively time consuming and expensive. We explore a fast and inexpensive way of doing it using Amazon's Mechanical Turk to pay small sums to a large number of non-expert annotators. For $10 we redundantly recreate judgments from a WMT08 translation task. We find that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality, and correlate more strongly with expert judgments than Bleu does. We go on to show that Mechanical Turk can be used to calculate human-mediated translation edit rate (HTER), to conduct reading comprehension experiments with machine translation, and to create high quality reference translations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Towards a Better Metric for Evaluating Question Generation Systems (2018, methodology): “e metrics are very popular and are used for a wide range of NLG tasks including AQG, there has always been criticism for using these metrics (for example, see (CallisonBurch et al.,2006;R et al.,2007;Callison-Burch, 2009)). More recently, there has been criticism (Liu et al.,2016) for using such met”
Fully Character-Level Neural Machine Translation without Explicit Segmentation (2016, background): “It is well known that automatic evaluation metrics such as BLEU encourage reference-like translations and do not fully capture true translation quality (Callison-Burch, 2009; Graham et al., 2015).”
Automatically Neutralizing Subjective Bias in Text (2019, background): “39%) on the same task in (Recasens, Danescu-Niculescu-Mizil, and Jurafsky 2013) (and can struggle on other tasks requiring linguistic knowledge (Callison-Burch 2009)).”
See No Evil, Say No Evil: Description Generation from Densely Labeled Images (2014, methodology): “We collected annotations of images in three stages using MTurk, and assigned each annotation task to 3-5 workers to improve quality through redundancy (Callison-Burch, 2009).”
Collecting Highly Parallel Data for Paraphrase Evaluation (2011, background): “…rapid data collection for various NLP tasks (Ambati and Vogel, 2010; Bloodgood and Callison-Burch, 2010a; Bloodgood and CallisonBurch, 2010b; Irvine and Klementiev, 2010), including human evaluations of NLP systems (CallisonBurch, 2009; Denkowski and Lavie, 2010; Zaidan and Callison-Burch, 2009).”
Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation (2018, unknown): “Combined ratings of bilingual crowd workers have been shown to be more reliable than automatic metrics and “very similar” to ratings produced by “experts”2 (Callison-Burch, 2009).”
Annotating Named Entities in Twitter Data with Crowdsourcing (2010, background): “This work details how to efﬁciently annotate large volumes of data, for information extraction tasks, at low cost using MTurk (Snow et al., 2008; Callison-Burch, 2009).”
BabelNet: Building a Very Large Multilingual Semantic Network (2010, methodology): “We will perform a semi-automatic validation of BabelNet, e.g. by exploiting Amazon’s Mechanical Turk (Callison-Burch, 2009) or designing a collaborative game (von Ahn, 2006) to validate low-ranking mappings and translations.”
Findings of the 2012 Workshop on Statistical Machine Translation (2012, background): “This year, we also opened up the evaluation to non-expert anno-tators hired on Amazon Mechanical Turk (Callison-Burch, 2009).”
Collecting Image Annotations Using Amazon’s Mechanical Turk (2010, methodology): “Related work MTurk has been used for many different NLP and vision tasks (Tietze et al., 2009; Zaidan and Callison-Burch, 2009; Snow et al., 2008; Sorokin and Forsyth, 2008).”
Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon (2010, methodology): “Amazon has an online service called Mechanical Turk that can be used to obtain a large amount of human annotation in an efficient and inexpensive manner (Snow et al., 2008; Callison-Burch, 2009).1 However, one must define the task carefully to obtain annotations of high quality.”
CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON (2013, methodology): “It is already being used to obtain human annotation on various linguistic tasks (Snow et al. , 2008; Callison-Burch, 2009).”
How Much Spam Can You Take? An Analysis of Crowdsourcing Results to Increase Accuracy (2011, background): “2 Worker error-rates Several studies have reported successes using different approaches to determine consensus [9,10,2].”