It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks

By David J. Hauser, Norbert Schwarz. 2015

170 citations recorded by the source corpus.

Instructional manipulation checks (IMCs) have become popular tools for identifying inattentive participants in online studies. IMCs function by attempting to trick inattentive participants into responding incorrectly. However, from a conversational perspective, question characteristics are part of the researcher’s contribution to the conversation, and IMCs may teach participants that there is “more than meets the eye,” prompting systematic thinking on subsequent tricky-seeming questions in an attempt to avoid being tricked. In two online studies, participants responded to a simple task either before or after completing an IMC. As expected, answering an IMC prior to the task improved performance on items that benefit from increased systematic thinking—namely, the Cognitive Reflection Test (Study 1), and a probabilistic reasoning task (Study 2). We conclude that IMCs change attention rather than merely measure attention and discuss implications for their use in online studies.

Subfields: Applied Psychology, Literature and Literary Theory, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/0a2e1cf5d310a08a0c7f6f925eba4457836db384

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Literature and Literary Theory

In Sociology and Political Science

Citations in the corpus

Conducting Clinical Research Using Crowdsourced Convenience Samples. (2016, background): “…recent years, the rate at which workers correctly answer “trick” questions that check for attentiveness has increased from a level similar to that observed among undergraduate samples (Goodman et al. 2013, Paolacci et al. 2010) to far above that of undergraduate samples (Hauser & Schwarz 2015a).”
Crowdsourcing Consumer Research (2017, background): “…motivated to follow instructions and pay attention to the research study (e.g., carefully consider a stimulus before answering the questions that follow), especially if they are aware of tests that may check their attention (Hauser and Schwarz 2015, 2016; Oppenheimer, Meyvis, and Davidenko 2009).”
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use (2018, background): “Some evaluations have found that MTurk participants’ attention is equal to or better than undergraduate participants’ attention (Chandler & Shapiro, 2016; Hauser & Schwarz, 2015, 2016).”
Sport spectatorship and life satisfaction: A multicountry investigation (2017, unknown): “The attention filters are designed to reduce noise associated with inattentive responses to online surveys (Hauser & Schwarz, 2015).”
Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations (2017, unknown): “Third, other research suggests that MTurk participants are just as accurate in their ability to recall previously seen items and are more attentive to instructions than undergraduate participants (Hauser and Schwarz, 2015; Ramsey et al., 2016).”
Are Attention Check Questions a Threat to Scale Validity? (2018, methodology): “However, it is noteworthy that studies demonstrated that the effect of attention checks on increased deliberation does not depend on the familiarity of attention checks (Hauser & Schwarz, 2015), which seems to suggest that experience in survey taking may not be an issue.”
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results (2019, background): “Similarly, instructional manipulation checks (IMCs) that were previously recommended as a way to ensure high-quality data (see Oppenheimer, Meyvis, & Davidenko, 2009) have been found to alter participants’ responses in potentially problematic ways (see Hauser & Schwarz, 2015).”
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments (2017, background): “There is also evidence that IMCs can lead to more systematic thinking (Hauser & Schwarz, 2015), though two studies did not find any increases in social desirability concerns or attrition rates (Berinsky et al., 2013; Clifford & Jerit, 2015).”
The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers (2025, unknown): “…rather than statements improves logical discernment [24], questions also improve critical reading [110, 142], attention checks promote systematic thinking [49], conflict-filled discussion induces critical thinking [78], and in general increased engagement results in behavioural changes [82, 92].”