No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments

By John V. Kane, Jason Barabas. American Journal of Political Science, 2018

224 citations recorded by the source corpus.

Manipulation checks are often advisable in experimental studies, yet they rarely appear in practice. This lack of usage may stem from fears of distorting treatment effects and uncertainty regarding which type to use (e.g., instructional manipulation checks [IMCs] or assessments of whether stimuli alter a latent independent variable of interest). Here, we first categorize the main variants and argue that factual manipulation checks (FMCs)—that is, objective questions about key elements of the experiment—can identify individual-level attentiveness to experimental information and, as a consequence, better enable researchers to diagnose experimental findings. We then find, through four replication studies, little evidence that FMC placement affects treatment effects, and that placing FMCs immediately post-outcome does not attenuate FMC passage rates. Additionally, FMC and IMC passage rates are only weakly related, suggesting that each technique identifies different sets of attentive subjects. Thus, unlike other methods, FMCs can confirm attentiveness to experimental protocols. Replication Materials: The data, code, and any additional materials required to replicate all analyses in this article are available on the American Journal of Political Science Dataverse within the Harvard Dataverse Network, at: https://dataverse.harvard.edu/dataset.xhtml?persistentId=https://doi.org/10.7910/DVN/GSQG1K. Experiments are a popular method for testing hypotheses and advancing theory within the social sciences. Essential for the validity of an experiment is the extent to which a study’s participants are actually “treated” (Oppenheimer, Meyvis, and Davidenko 2009). As a precondition for being treated, respondents must (in most instances) first be attentive to treatments. Ensuring that respondents attend to stimuli, however, presents a challenge, particularly given the growth in online surveys and reliance upon subjects from opt-in panels who may be distracted (e.g., Chandler and Shapiro 2016; Clifford and Jerit 2015). Within a study, therefore, being able to accurately gauge receipt of treatment is paramount. One technique for assessing treatment receipt is that of a manipulation check (MC). Broadly defined, MCs are “used to check whether the manipulation conducted in an experiment is perceived by the subjects as the experimenter wishes it to be perceived” (Morton and John V. Kane is Assistant Professor, Center for Global Affairs, New York University, 15 Barclay Street, New York, NY 10007 (john.kane@nyu.edu).Jason Barabas is Professor, Department of Political Science, Stony Brook University, Social & Behavioral Sciences, 7th Floor, Stony Brook, NY 11794 (jason.barabas@stonybrook.edu). We would like to thank Scott Clifford, Jennifer Jerit, Yanna Krupnikov, David Nickerson, and Christine Peterson for their helpful comments on earlier drafts of this article. Additionally, Ben Carter, Mike Kriner, Michelle lo-Low, and Alecia Nepaul provided outstanding research assistance. Matthew Hitt and Vittorio Mérola graciously provided data and code. Finally, we also appreciate the suggestions that we received from the anonymous reviewers, faculty and graduate students at Stony Brook University, and panelists at the annual conferences of the American Political Science Association and the International Society for Political Psychology. Williams 2010, 108). However, there is much variation in why researchers implement MCs. In practice, scholars sometimes report using manipulation checks to determine whether the latent independent variable of interest has been affected by experimental stimuli (Mutz and Pemantle 2015, 196). More generally, researchers also use MCs to assess respondent attentiveness during a study, such as via specific questions given to participants to judge whether they are reading carefully (e.g., Anduiza and Galais 2016; Crump, McDonnell, and Gureckis 2013; Maniaci and Rogge 2014). MCs therefore provide researchers with leverage on the nature of their experimental findings. As Mutz (2011, 84–85) contends, “Experiments for which manipulation checks can be considered ‘optional’ are relatively few and far between,” primarily because failing to include an MC “undermines what can be learned from any given study.” Perhaps because of the varying functions of MCs, there is widespread heterogeneity in the forms that MCs American Journal of Political Science, Vol. 63, No. 1, January 2019, Pp. 234–249 C ©2018, Midwest Political Science Association DOI: 10.1111/ajps.12396

Subfields: Applied Psychology, Cognitive Neuroscience, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/1ce00d50b15411208cff5116dcd18803a9354c73