The labor economics of paid crowdsourcing
By John Joseph Horton, Lydia B. Chilton. 2010
677 citations recorded by the source corpus.
We present a model of workers supplying labor to paid crowdsourcing projects. We also introduce a novel method for estimating a worker's reservation wage - the key parameter in our labor supply model. We tested our model by presenting experimental subjects with real-effort work scenarios that varied in the offered payment and difficulty. As predicted, subjects worked less when the pay was lower. However, they did not work less when the task was more time-consuming. Interestingly, at least some subjects appear to be "target earners," contrary to the assumptions of the rational model. The strongest evidence for target earning is an observed preference for earning total amounts evenly divisible by 5, presumably because these amounts make good targets. Despite its predictive failures, we calibrate our model with data pooled from both experiments. We find that the reservation wages of our sample are approximately log normally distributed, with a median wage of $1.38/hour. We discuss how to use our calibrated model in applications.
Subfields: Computer Science Applications, Management Science and Operations Research, Safety Research, Natural Language Processing, Economics & Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Science Applications
- Gabriele Paolacci
- J. Goodman
- Jesse J. Chandler
- Adam Marcus
- Aditya G. Parameswaran
- D. Shapiro
- Michael S. Bernstein
- Dana Chandler
In Management Science and Operations Research
- Adam Marcus
- Adish Kumar Singla
- Aditya G. Parameswaran
- Andreas Krause
- Dana Chandler
- Jaime Teevan
- Matthew Richardson
- Irwin King
In Safety Research
- Gabriele Paolacci
- J. Goodman
- Jesse J. Chandler
- Nicolas Sommet
- Martin J. Burnham
- Yen K. Le
- Dana Chandler
- Andreas Krause
In Natural Language Processing
- Yuxiang (Chris) Zhao
- Qinghua Zhu
- Matthew Richardson
- Dana Chandler
- Jaime Teevan
- Irwin King
- Andreas Krause
- Daniel J. Veit
In Economics & Operations Research
- Dana Chandler
- Andreas Krause
- Daniel J. Veit
- J. Goodman
- A. Elliot
- Adish Kumar Singla
- Kevin Crowston
- Jennifer Widom
Citations in the corpus
- MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text (2013, background): “We did not experiment with paying less, but this rate amounts to $6.82/hour, which is approximately the rate paid by other writing tasks on AMT at the time, though is also significantly higher than the median wage of $1.38 found in 2010 (Horton and Chilton, 2010).”
- Crowdsourced Data Management: Industry and Academic Perspectives (2015, background): “” In particular, the community has studied incentive structures in crowdsourcing marketplaces [96, 52]; they have also studied how to improve the efficiency of crowdsourcing [110], games with a purpose [103, 102], crowdsourcing contests [50], Question-Answer (QA) forums [101], and user-generated con”
- Achievement Goals, Reasons for Goal Pursuit, and Achievement Goal Complexes as Predictors of Beneficial Outcomes: Is the Influence of Goals Reducible to Reasons? (2017, positive): “3 For this and the subsequent studies, the payment was way well above the reservation wage of USD 1.38 per hour (i.e., the minimum wage a worker is willing to accept to complete a task; Horton & Chilton, 2010).”
- Fair Work: Crowd Work Minimum Wage with One Line of Code (2019, background): “…majority of workers are honest and will report honestly or with only slight infla-tion (Ariely 2008; Hancock, Toma, and Ellison 2007), and that even slight inflation could be considered compensation for the considerable unpaid time that workers spend searching for tasks (Horton and Chilton 2010).”
- A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk (2017, background): “minimum wage [32,33,35,36,37,47] and many workers report not being paid for adequately completed tasks [36,49].”
- Using Mechanical Turk to Study Clinical Populations (2013, positive): “First, data can be collected quickly for a minimal cost (Horton & Chilton, 2010).”
- Who is Mturk? Personal characteristics and sample consistency of these online workers (2018, background): “Some researchers have noted demographic differences between MTurkers and the US population on age (Bohannon, 2016; Horton & Chilton, 2010; Mason & Suri, 2011) and gender (Mason & Suri, 2011); between MTurkers and internet samples on age (Buhrmester et al., 2011); and between MTurkers and a US…”
- Conducting Clinical Research Using Crowdsourced Convenience Samples. (2016, background): “Horton & Chilton (2010) identified an average wage of $1.38/hour on MTurk, although this study included international workers.”
- Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers (2018, background): “Low per-task re-wards and unpaid task search time contribute to more than half of the AMT workers currently earning less than $5 an hour (Ipeirotis 2010; Irani and Silberman 2013; 2016; Hitlin 2016; Horton and Chilton 2010; Martin et al. 2014; Hara et al. 2018).”
- Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment (2015, neutral): “38/hour [23].”
- Stimulating Online Reviews by Combining Financial Incentives and Social Norms (2017, positive): “For example, Horton and Chilton (2010) asked Turkers to click on a pair of rectangles in a specific order, offering them payment for each completed series of clicks.”
- The Future of Work in the ‘Sharing Economy’. Market Efficiency and Equitable Opportunities or Unfair Precarisation? (2016, neutral): “(Horton & Chilton, 2010) Labour economics of crowdsourced work Economics Quantitative study Experiment MTurk 36.”
- Evaluation on crowdsourcing research: Current status and future direction (2012, background): “…important in many IS areas, and some studies have focused on the incentive aspects of crowdsourcing, especially from the economics which looks at incentive issues from a game theoretic perspective (Archak and Sundararajan 2009; DiPalantino and Vojnovic 2009; Horton and Chilton 2010; Wilcox 2000).”
- CrowdFill: collecting structured data from the crowd (2014, methodology): “While reference [12] describes a method for estimating a worker’s reservation wage (i.e., the lowest wage a worker is willing to accept), more recent studies [20, 21] propose an alternative approach where workers indicate their desired wage by submitting a bid.”
- Inside the Turk (2014, background): “Interestingly, workers appear to be sensitive to nominal pay rates, finding tasks that pay in multiples of 5 cents to be more attractive (Horton & Chilton, 2010).”
- Online Task Assignment in Crowdsourcing Markets (2012, background): “…to inex-pensively obtain distributed labor and information, and have recently become popular among researchers who use sites like Mechanical Turk to conduct user studies (Kittur, Chi, and Suh 2008), run behavioral experiments (Mason and Suri 2012), and collect data (Horton and Chilton 2010).”
- A Survey of Crowdsourcing Systems (2011, background): “To balance between a companys production cost and its workers’ reservation wage, Horton and Chilton [31] presented a labor supply model to estimate a worker’s reservation wage.”