“Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents

By C. Huff, Dustin Tingley. 2015

As Amazon’s Mechanical Turk (MTurk) has surged in popularity throughout political science, scholars have increasingly challenged the external validity of inferences made drawing upon MTurk samples. At workshops and conferences experimental and survey-based researchers hear questions about the demographic characteristics, political preferences, occupation, and geographic location of MTurk respondents. In this paper we answer these questions and present a number of novel results. By introducing a new benchmark comparison for MTurk surveys, the Cooperative Congressional Election Survey, we compare the joint distributions of age, gender, and race among MTurk respondents within the United States. In addition, we compare political, occupational, and geographical information about respondents from MTurk and CCES. Throughout the paper we show several ways that political scientists can use the strengths of MTurk to attract respondents with specific characteristics of interest to best answer their substantive research questions.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Diagnosing Gender Bias in Image Recognition Systems (2020, methodology): “Although “MTurkers” have often been a population sampled for survey research (Huff and Tingley 2015), these workers have also been employed to assist in the research process itself (Shank 2016), as was the case for our project.”
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection (2021, background): “12 With the task at hand, we sought a racially and politically diverse pool of participants, which can be challenging given that MTurk workers are usually tend to be predominantly white and skew liberal (Huff and Tingley, 2015; Burnham et al., 2018; Loepp and Kelly, 2020).”
Contraction as a Response to Group Threat: Demographic Decline and Whites’ Classification of People Who Are Ambiguously White (2020, background): “MTurk workers are not representative of the U.S. population (Huff and Tingley 2015; Paolacci and Chandler 2014).”
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments (2017, background): “…Berinsky, Huber, & Lenz, 2012; Buhrmester, Kwang,& Gosling, 2011; Chandler, Mueller,& Paolacci, 2014; Goodman, Cryder, & Cheema, 2012; Horton, Rand, & Zeckhauser, 2011; Huff & Tingley, 2015; Leeper, 2016; Mason & Suri, 2012; Paolacci, Chandler, & Ipeirotis, 2010; Rand, 2012; Suri & Watts, 2011).”
Conducting Clinical Research Using Crowdsourced Convenience Samples. (2016, background): “Interestingly, measures of household income are closer to national averages (Greenblatt 2013, Huff & Tingley 2014) and workers are less likely to live alone (Greenblatt 2013, 2014).”
Exploring User Perceptions of Discrimination in Online Targeted Advertising (2017, background): “1 MTurk has been shown to provide adequate data quality, but also to be younger and more educated than the general population [24, 26].”
An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk (2017, background): “Some large-scale, survey-based studies have found that MTurk respondents are not too different from respondents on other survey platforms (Huff and Tingley 2015), but others have identiﬁed some differences in age, education, gender, and income (Maynard 2014; Peer et al. 2015).”
VLAT: Development of a Visualization Literacy Assessment Test (2017, background): “The workers well represent the US population [6, 27].”
Influence of brand relationship on customer attitude toward integrated resort brands: a cognitive, affective, and conative perspective (2018, result): “…has suggested that recruiting respondents from Amazon Mechanical Turk is similar to respondents on other survey platforms, and that the Amazon Mechanical Turk population is appreciably representative of the United States (US) population (Berinsky, Huber, & Lenz, 2012; Huff & Tingley, 2015).”
Crowdsourcing research: Data collection with Amazon’s Mechanical Turk (2018, background): “Early studies of MTurk showed a predominance of men in the pool; yet, more recent studies show an almost equal representation of men and women (Chambers, Nimon, & Anthony-McMann, 2016; Hitlin, 2016; Huff & Tingley, 2015).”