The Language Demographics of Amazon Mechanical Turk

By Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, Chris Callison-Burch. 2014

120 citations recorded by the source corpus.

We present a large scale study of the languages spoken by bilingual workers on Mechanical Turk (MTurk). We establish a methodology for determining the language skills of anonymous crowd workers that is more robust than simple surveying. We validate workers’ self-reported language skill claims by measuring their ability to correctly translate words, and by geolocating workers to see if they reside in countries where the languages are likely to be spoken. Rather than posting a one-off survey, we posted paid tasks consisting of 1,000 assignments to translate a total of 10,000 words in each of 100 languages. Our study ran for several months, and was highly visible on the MTurk crowdsourcing platform, increasing the chances that bilingual workers would complete it. Our study was useful both to create bilingual dictionaries and to act as census of the bilingual speakers on MTurk. We use this data to recommend languages with the largest speaker populations as good candidates for other researchers who want to develop crowdsourced, multilingual technologies. To further demonstrate the value of creating data via crowdsourcing, we hire workers to create bilingual parallel corpora in six Indian languages, and use them to train statistical machine translation systems.

Subfields: Artificial Intelligence, Computer Science Applications, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Evaluation & Benchmarks

Citations in the corpus

Neural Generation for Czech: Data and Baselines (2019, methodology): “Since there are few to none Czech speakers on crowdsourcing platforms (Pavlick et al., 2014; Duˇsek et al., 2014), we were not able to use them for data collection.”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “This feedback was done only for English tasks, as finding high-quality crowdworkers in other languages is nontrivial (Pavlick et al., 2014).”
The Social Impact of Natural Language Processing (2016, background): “2Héctor Martı́nez Alonso, personal communication 3Except for annotation: there are a number of papers on the status of crowdsource workers (Fort et al., 2011; Pavlick et al., 2014).”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, methodology): “…government websites (Haddow and Kirefu, 2020; Siripragada et al., 2020), the Tanzil corpus (Tiedemann, 2009), the Pavlick dic- 4 5 6 7 tionaries (Pavlick et al., 2014), a corpus3 produced 3https://github.com/nlpcuom/ by the University of Moratuwa, the HindEnCorp…”
Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics (2018, background): “age of workers relied on MTurk to make basic ends meet. Later studies on the demographics of MTurk workers reported similar results (e.g. Goodman and Paolacci (2017); Berg (2016); Peer et al. (2017); Pavlick et al. (2014); Naderi (2018); Difallah et al. (2018)). On MTurk, American and Indian crowdwo”
Text-to-Speech Synthesis Using Found Data for Low-Resource Languages (2019, background): “(Pavlick et al., 2014) has revealed a linguistically-diverse community of workers on Mechanical Turk – a sample of about 3,200 bilingual workers revealed native fluency in over 80 different languages, 35 of which had more 14 than 20 speakers, and 9 of which are languages for which we have both read…”
Opportunities and Challenges for Artificial Intelligence in India (2018, background): “In a recent survey, Pavlick et al. (2014) identify languages that are good bets for linguists to study, since they can find translators on Amazon Mechanical Turk.”
PMIndia - A Collection of Parallel Corpora of Languages of India (2020, methodology): “For hunalign, we used the crowd-sourced dictionaries from (Pavlick et al., 2014), arbitrarily selecting the first translation where there was more than one.”
Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks (2022, background): “This feedback was done only for English tasks, as finding high-quality crowdworkers in other languages is non-trivial (Pavlick et al., 2014).”
Five sources of bias in natural language processing (2021, background): “For example, various works have shown that crowdsourced annotators' demographic makeup is not as representative as one might hope (Pavlick et al., 2014).”
Turkers, Scholars, "Arafat" and "Peace": Cultural Communities and Algorithmic Gold Standards (2015, background): “Speciﬁcally, we show that AMT workers, communities of general academics and subject-area experts each have their own “gold standard” for a series of SR problem instances.”
Exploring Stereotypes and Biased Data with the Crowd (2018, background): “Pavlick et al. (2014) performed a large scale study of the language abilities of the workers on AMT.”
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, methodology): “We also provide a Nepali–English bilingual dictionary with 9,916 entries (Pavlick et al., 2014).”