Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample

By Jill D. Weinberg, J. Freese, David McElhattan. 2014

533 citations recorded by the source corpus.

Compared to older kinds of sample surveys, online platforms provide a fast and low-cost platform for factorial surveys, as well as a more demographically diverse alternative to student samples. Two distinct strategies have emerged for recruitment: using panels based on population-based samples versus recruiting people actively seeking to complete online tasks for money. The latter is much cheaper but prompts various concerns about data quality and generalizability. We compare results of three vignette experiments conducted using the leading online panel that uses a population-based paradigm (Knowledge Networks, now GfK) and the leading platform for crowdsource recruitment (Amazon Mechanical Turk). Our data show that, while demographic differences exist, most notably in age, the actual results of our experiments are very similar, especially once these demographic differences have been taken into account. Indicators of data quality were actually slightly better among the crowdsource subjects. Although more evidence is plainly needed, our results support the accumulating evidence for the promise of crowdsource recruitment for online experiments, including factorial surveys.

Subfields: Computer Science Applications, Safety Research, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/4bb8aae46433447ac8f9c0f39065e9e0ec334907

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Safety Research

In Sociology and Political Science

Citations in the corpus

Conducting interactive experiments online (2017, unknown): “7 In comparison to the student or community samples normally used in the laboratory, Mturk samples are also more diverse (Krupnikov and Levine 2014; Paolacci and Chandler 2014; Weinberg et al. 2014; Berinsky et al. 2012).”
Conducting Clinical Research Using Crowdsourced Convenience Samples. (2016, background): “…MTurk samples are less representative than web-based probability samples (i.e., individuals recruited through random digit dialing and invited to participate in an online panel) in terms of gender, race, income, and marital status (Berinsky et al. 2012, Mullinix et al. 2014, Weinberg et al. 2014).”
Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics (2018, background): “Weinberg et al. (2014) analyzed socio-demographic characteristics of workers on MTurk and compared them to the characteristics of respondents of a population-based web panel.”
How Black Are Lakisha and Jamal? Racial Perceptions from Names Used in Correspondence Audit Studies (2017, unknown): “There is some evidence that these demographic differences account for minimal differences in effect sizes between MTurk and other Internet survey platforms that claim representative samples (Weinberg et al. 2014).”
Advantages of Matched Over Unmatched Opt-in Samples for Studying Criminal Justice Attitudes: A Research Note (2020, methodology): “These are convenience samples (Landers & Behrend, 2015), but they provide a relatively inexpensive method of exploring many theories and ideas and have the potential to yield a diverse, nationwide group of respondents (Weinberg et al., 2014).”
The shape of and solutions to the MTurk quality crisis (2020, background): “…more diverse than most common convenience samples and tend to replicate a variety of experimental and observational results (Berinsky et al., 2012; Weinberg et al., 2014; Clifford et al., 2015; Mullinix et al., 2015).1 Though met with skepticism by some, MTurk respondents tend to yield…”
The Generalizability of Survey Experiments* (2015, background): “…has sought to compare samples (e.g. Berinsky et al. 2012; Goodman et al. 2012; Horton et al. 2011; Krupnikov and Levine 2014; Paolacci et al. 2010; Weinberg et al. 2014).3 While these studies are impressive and telling, each includes only a small number of comparisons (e.g. three experiments) on…”