Who are the crowdworkers?: shifting demographics in mechanical turk

By Joel Ross, Lilly C. Irani, Michael Six Silberman, Andrew Zaldivar, Bill Tomlinson. CHI Extended Abstracts, 2010

1,245 citations recorded by the source corpus.

Subfields: Computer Science Applications, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/2f845a587699129bdd130e567a25785bcb5e97af

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Management Science and Operations Research

Citations in the corpus

A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality (2016, background): “worker population. Ages 18-29, 50-64, and 65+ respectively comprised 19:0%, 23:3% and 3:6% of the long-term worker population. Compared to the distributions in previously gathered demographics on AMT [11,17,47], the gender and age distribution of all workers closely aligns with these other previousl”
A Survey on Mobile Crowdsensing Systems: Challenges, Solutions, and Opportunities (2019, background): “Several studies consider Amazon’s Mechanical Turk as one of the most important examples of incentive mechanisms for micro-tasks, the paradigm allowing to engage a multitude of users for short time and low monetary cost [90], [91].”
Analyzing the Amazon Mechanical Turk marketplace (2010, background): “For example, a common set of questions that pop up in an “introduction to crowdsourcing and AMT” session are the following: • Who are the workers that complete these tasks? • What type of tasks can be completed in the marketplace? • How much does it cost? • How fast can I get results back? • How big”
Predicting the brighter and darker sides of interpersonal relationships: Does psychological need thwarting matter? (2015, result): “This ethnic distribution is in line with previous research recruiting participants from this website (Paolacci et al. 2010; Ross et al. 2010; Mason and Suri 2012).”
Virtual and Real World Adaptation for Pedestrian Detection (2014, background): “Index Terms—Pedestrian detection, photo-realistic computer animation, data set shift, domain adaptation Ç”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, background): “Majority vote also necessarily obfus-cates differences among groups underrepresented in annotator pools, such as older adults who can exhibit views on aging distinct from crowd workers (Díaz, 2020), the majority of whom tend to be younger (Ross et al., 2010).”
Implicit Bias in Crowdsourced Knowledge Graphs (2019, background): “Crowdsourcing platforms like Amazon MTurk enable access to workers having a large diversity of cultural backgrounds [18].”
There's No Such Thing as the Perfect Map: Quantifying Bias in Spatial Crowd-sourcing Datasets (2015, result): “As an example, a recent study has revealed that, in 2008, power users on Amazon Mechanical Turk were primarily American, young, female, and well-educated; however, only one year after, a fundamental change had taken place, and the majority of the Turkers had then become male Indians [26].”
Serf and turf: crowdturfing for fun and profit (2011, background): “nnectivity. 7. RELATED WORK Crowd-sourcing Research. Since coming online in 2005, Amazon’s Mechanical Turk has been scrutinized by the research community. This includes studies of worker demographics [13, 27], task pricing [6, 12], and even meta-studies on how to useMechanical Turk to conduct user s”
Flexibility in the Gig Economy: Managing Time on Three Online Piecework Platforms (2018, neutral): “…class women supplementing their husbands’ incomes (Benería & Roldán 1987, Boris & Daniels 1989), while online piecework is not associated with any particular social group; surveys suggest that it is done across genders, income classes, and educational backgrounds (Ross 2010, Hitlin et al. 2016).”
We get the algorithms of our ground truths: Designing referential databases in digital image processing (2017, background): “For a study of the demographics of crowd workers, see Ross et al. (2010).”
Scoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets (2017, methodology): “…that workers can earn from $8 to $16 per hour by completing many tasks (e.g., Chen 2012; Roggio 2012), while both academic literature and media reports suggest wages of $1.20 to $5.00 per hour, or even less (e.g., Ross et al. 2010; Buhrmester, Kwang and Gosling 2011; Dobson 2013; Folbre 2013).”
Crowdsourcing Consumer Research (2017, result): “In the largest recent demographic survey of MTurk (nearly 10,000 workers), Casey et al. 2017 found results consistent with previous similar investigations (Berinsky, Huber, and Lenz 2012; Paolacci, Chandler, and Ipeirotis 2010; Ross et al. 2010).”