The viability of crowdsourcing for survey research

By Tara S. Behrend, David J. Sharek, Adam W. Meade, Eric N. Wiebe. Behavior Research Methods, 2011

1,107 citations recorded by the source corpus.

Online contract labor portals (i.e., crowdsourcing) have recently emerged as attractive alternatives to university participant pools for the purposes of collecting survey data for behavioral research. However, prior research has not provided a thorough examination of crowdsourced data for organizational psychology research. We found that, as compared with a traditional university participant pool, crowdsourcing respondents were older, were more ethnically diverse, and had more work experience. Additionally, the reliability of the data from the crowdsourcing sample was as good as or better than the corresponding university sample. Moreover, measurement invariance generally held across these groups. We conclude that the use of these labor portals is an efficient and appropriate alternative to a university participant pool, despite small differences in personality and socially desirable responding across the samples. The risks and advantages of crowdsourcing are outlined, and an overview of practical and ethical guidelines is provided.

Subfields: Computer Science Applications, Information Systems and Management, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/81d50f350a63f5e7be0058e87e7d60f12b9af3b2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems and Management

In Sociology and Political Science

Citations in the corpus

Detecting and Deterring Insufficient Effort Responding to Surveys (2012, background): “Researchers and practitioners have expressed a growing interest in using certain indices for careless responding to ensure the quality of their survey data (e.g., Behrend et al. 2011; Curran et al. 2010; Johnson 2005; Meade and Craig 2011).”
An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other Convenience Samples (2015, background): “3 See Behrend et al. (2011) for an example of how these comparisons could be conducted.”
Conducting interactive experiments online (2017, neutral): “(2010), Behrend et al. (2011), Klein et al. (2014), and Mullinix et al. (2015). 3 Thus far interactive research developed its own bespoke software (see: Egas and Riedl 2008; Suri and Watts 2011; Wang et al.”
Detecting Insufficient Effort Responding with an Infrequency Scale: Evaluating Validity and Participant Reactions (2014, background): “MTurk, a crowdsourcing service that offers cost-effective access to a large number of respondents online (Behrend et al. 2011), has been shown to provide”
Dirty Data: The Effects of Screening Respondents Who Provide Low-Quality Data in Survey Research (2017, background): “…(Behrend, Sharek, Meade, &Wiebe, 2011; Berry et al., 1992), the number of consecutive identical responses provided by the respondent (“longstring”; Behrend et al., 2011; Huang et al., 2012; Meade & Craig, 2012), and individual response variability (IRV; Dunn, Heggestad, Shanock, & Nels, in…”
Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations (2017, background): “MTurk has also demonstrated measurement equivalence with other samples including student and employee samples (Behrend et al., 2011; Steelman et al., 2014; Feitosa et al., 2015).”
Computational Sociolinguistics: A Survey (2016, methodology): “te their answers towards what they think is expected, especially when there is a monetary compensation. In the social sciences in general, crowdsourcing is also increasingly used for survey research. Behrend et al. (2011)compared the data collected using crowdsourcing with data collected from a trad”
Don't Throw the Baby Out With the Bathwater: Comparing Data Quality of Crowdsourcing, Online Panels, and Student Samples (2015, background): “Crowdsourcing (e.g., MTurk) represents a fast and inexpensive sampling method, allowing researchers to collect data from quite large samples in only hours for as cheap as a few cents or dollars per participant (Behrend et al., 2011).”