A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

By Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, Jeffrey P. Bigham. 2017

484 citations recorded by the source corpus.

A growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk. Our task-level analysis revealed that workers earned a median hourly wage of only ~$2/h, and only 4% earned more than $7.25/h. While the average requester pays more than $11/h, lower-paying requesters post much more work. Our wage calculations are influenced by how unpaid work is accounted for, e.g., time spent searching for tasks, working on tasks that are rejected, and working on tasks that are ultimately not submitted. We further explore the characteristics of tasks and working patterns that yield higher hourly wages. Our analysis informs platform design and worker tools to create a more positive future for crowd work.

Subfields: Artificial Intelligence, Computer Science Applications, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, neutral): “…much crowdworkers actually earn vary, but generally show an average lower than minimum wage [30]; surveys of workers from Amazon Mechanical Turk and Crowdﬂower place it on average be-tween $1 and $5.5 per hour [7] with a median wage of roughly $2 / hour [24, 52]; only a small fraction of workers (4”
It's Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process (2021, background): “Crowdwork, however, also comes with serious negative externalities such as incentivizing very low pay [18].”
Fair Work: Crowd Work Minimum Wage with One Line of Code (2019, background): “Microtask crowdsourcing remains a precarious income source for crowd workers (Gray and Suri 2019; Kittur et al. 2013; McInnis et al. 2016; Martin et al. 2014; Hara et al. 2018).”
Conceptualizing the Gig Economy and Its Regulatory Problems (2020, background): “Some scholars avoid a general definition, instead focusing on a specific platform (Birgillito & Birgillito, 2018; De Groen et al., 2016; Green et al., 2018; Hara et al., 2018) or a specific sector (Cramer & Krueger, 2016).”
A Survey of NLP-Related Crowdsourcing HITs: what works and what does not (2021, background): “Even though rejections constitute a small percentage of work (Hara et al., 2018), real workers who have honestly put effort into their work should be paid for that time.”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “Critics argue that these platforms are largely unregulated and that workers are poorly compensated [91, 224].”
Rehumanized Crowdsourcing: A Labeling Framework Addressing Bias and Ethics in Machine Learning (2019, background): “These have serious consequences because many depend on doing tasks to secure their income [30], which leads to more competition for tasks, and more often than not, a small number of workers (i.e., the most active) submitting a large fraction of the work available [13].”
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices (2021, background): “Proper compensation can become a more frequent practice due to current features (like qualifications or badges) in crowdsourcing environments, advances in techniques to safeguard quality, plus under-payment issues being a reiterating topic addressed by recent literature [5, 33, 39, 79].”
Requiem for a Nudge: Framing effects in nudging honesty (2020, unknown): “The experiment lasted about 9 minutes and participants earned an average of $1.20 (including a show-up payment of $0.30), which translates to an hourly wage of roughly $8 and is well above average on the MTurk platform ( Hara et al., 2018 ).”
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning (2021, background): “We decide upon these numbers after a few timed test trials run by ourselves. we ﬁnd that these numbers enable the workers to acquire 6 USD per hour, an above average hourly wage on the AMT platform [15].”
The Expertise Involved in Deciding which HITs are Worth Doing on Amazon Mechanical Turk (2021, background): “The difficulty of finding enough fair-paying wages is further evident given that while 80% of all Turkers are based in the United States [24], only 4% of Turkers make above the United States minimum wage of $7.25 an hour [22].”
Disability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors (2021, background): “One could compare data collection to crowd work and consider the payment rates in that context [23].”
Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers (2018, background): “Low per-task re-wards and unpaid task search time contribute to more than half of the AMT workers currently earning less than $5 an hour (Ipeirotis 2010; Irani and Silberman 2013; 2016; Hitlin 2016; Horton and Chilton 2010; Martin et al. 2014; Hara et al. 2018).”