Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents

By A. Coppock, Oliver A. McClellan. Research & Politics, 2019

804 citations recorded by the source corpus.

Researchers have increasingly turned to online convenience samples as sources of survey responses that are easy and inexpensive to collect. As reliance on these sources has grown, so too have concerns about the use of convenience samples in general and Amazon’s Mechanical Turk in particular. We distinguish between “external validity” and theoretical relevance, with the latter being the more important justification for any data collection strategy. We explore an alternative source of online convenience samples, the Lucid Fulcrum Exchange, and assess its suitability for online survey experimental research. Our point of departure is the 2012 study by Berinsky, Huber, and Lenz that compares Amazon’s Mechanical Turk to US national probability samples in terms of respondent characteristics and treatment effect estimates. We replicate these same analyses using a large sample of survey responses on the Lucid platform. Our results indicate that demographic and experimental findings on Lucid track well with US national benchmarks, with the exception of experimental treatments that aim to dispel the “death panel” rumor regarding the Affordable Care Act. We conclude that subjects recruited from the Lucid platform constitute a sample that is suitable for evaluating many social scientific theories, and can serve as a drop-in replacement for many scholars currently conducting research on Mechanical Turk or other similar platforms.

Subfields: Communication, Sociology and Political Science, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/c39be909a707344b260eb08280d343a8862d1d98

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Communication

In Sociology and Political Science

In Statistical and Nonlinear Physics

Citations in the corpus

Designing Information Provision Experiments (2020, background): “Coppock and McClellan (2019) find that samples from Lucid score similarly to respondents in the American National Election Study (ANES) on the Big-5 personality inventory, show similar levels of political knowledge, and recover framing effects similar to the ones observed in a probability-based…”
A Practical Guide to Doing Behavioral Research on Fake News and Misinformation (2020, background): “The study was run on Lucid, which is a polling firm that collects U.S. samples that are quota-matched to national demographics on age, gender, region, and ethnicity (Coppock & McClellan, 2019).”
Fighting misinformation on social media using crowdsourced judgments of news source quality (2019, result): “In the second study, we tested the generalizability of our findings by surveying an additional n = 970 Americans recruited from Lucid, providing a subject pool that is nationally representative on age, gender, ethnicity, and geography (27).”
Factors Associated With US Adults’ Likelihood of Accepting COVID-19 Vaccination (2020, neutral): “(%) Age, y 18-29 456 (23) 30-44 577 (29) 45-59 490 (25) ≥60 448 (23)”
Correction Experiences on Social Media During COVID-19 (2021, background): “…a combination of screening questions and quota sampling on the basis of age, race, ethnicity, education, income, party affiliation, and region (Coppock & McClellan, 2019).2 A total of 1,094 people participated, and our sample characteristics were 51% male, 76% White (13% Black, 5% Asian, and…”
Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA (2023, background): “Worse data quality has also been found following the COVID-19 pandemic on Lucid [23], another data collection platform which had previously been recommended as a viable alternative to MTurk [24].”
Fighting COVID-19 Misinformation on Social Media: Experimental Evidence for a Scalable Accuracy-Nudge Intervention (2020, methodology): “We recruited 1,000 participants using Lucid, an online recruiting source that aggregates survey respondents from many respondent providers (Coppock & Mcclellan, 2019).”
On the belief that beliefs should change according to evidence: Implications for conspiratorial, moral, paranormal, political, religious, and science beliefs (2019, background): “…– a source that provides American samples that are nationally representative on age, gender, ethnicity, and geography (based on quota-matching), and that therefore provides a more even and representative split of liberals and conservatives (Coppock & Mcclellan, 2019; Pennycook & Rand, 2019a).”
Out of One, Many: Using Language Models to Simulate Human Samples (2022, methodology): “To formally analyze this data, we hired 2873 individuals through the survey platform Lucid [31] to evaluate the 7675 texts produced by human and GPT-3 survey respondents, without any indication of which was which.”
Emphasizing publishers does not effectively reduce susceptibility to misinformation on social media (2020, methodology): “We then assigned participants from MTurk and the sampling company Lucid (Coppock & McClellan, 2019) to either rate the trustworthiness of a random subset of the 60 sources (N=251 from MTurk in Study 4, N=253 from Lucid in Study 5) or rate the accuracy of a random subset of the 600 headlines…”
Human heuristics for AI-generated language are flawed (2022, methodology): “For the main experiment in the hospitality context, we recruited a US-representative sample of 2,000 participants through Lucid (48).”