Measuring and Bounding Experimenter Demand

By Jonathan de Quidt, J. Haushofer, Christopher Roth. The American Economic Review, 2018

440 citations recorded by the source corpus.

We propose a technique for assessing robustness to demand effects of findings from experiments and surveys. The core idea is that by deliberately inducing demand in a structured way we can bound its influence. We present a model in which participants respond to their beliefs about the researcher’s objectives. Bounds are obtained by manipulating those beliefs with “demand treatments.” We apply the method to 11 classic tasks, and estimate bounds averaging 0.13 standard deviations, suggesting that typical demand effects are probably modest. We also show how to compute demand-robust treatment effects and how to structurally estimate the model. (JEL C83, C90, D83, D91)

Subfields: Economics and Econometrics, General Decision Sciences, Safety Research

View Paper: https://www.semanticscholar.org/paper/9f84b7fa1e9c9d6b403cb6f48d71eefe8dbbf1b0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In General Decision Sciences

In Safety Research

Citations in the corpus

Designing Information Provision Experiments (2020, neutral): “One concern with information provision experiments are demand effects (de Quidt et al., 2018; Mummolo and Peterson, 2019; Zizzo, 2010).12 While recent empirical evidence suggests a limited quantitative importance of experimenter demand effects in online surveys in some domains (de Quidt et al.,…”
Parents’ Beliefs about Their Children’s Academic Ability: Implications for Educational Investments (2019, positive): “De Quidt et al. (2017) provide evidence that demand e ects are modest with incentivized choices.”
The Effect of Macroeconomic Uncertainty on Household Spending (2021, background): “This allows tracking both the immediate and more persistent effects of changes in uncertainty on household spending and helps rule out survey demand effects that have been shown to be small in settings like ours (De Quidt et al 2018).”
On making experimental design choices: Discussions on the use and challenges of demand effects, incentives, deception, samples, and vignettes (2020, methodology): “De Quidt, Haushofer, and Roth (2018) explore demand effects in economic games, and Mummolo and Peterson (2019) explore demand effects in vignette experiments.”
How Do Expectations about the Macroeconomy Affect Personal Expectations and Behavior? (2020, background): “28Evidence suggests that respondents in online surveys respond only very moderately to explicit signals about the experimenter’s wishes (de Quidt et al., 2018). reported expectations in a setting close to ours are not responsive to explicit signals about the experimental hypothesis.”
What Have We Learned about Gender from Candidate Choice Experiments? A Meta-Analysis of Sixty-Seven Factorial Survey Experiments (2021, background): “Such explanations are dif ﬁ cult to square, however, with recent investigations that have found very little evidence for demand effects, especially in the context of studies administered online (e.g., De Quidt, Haushofer, and Roth 2018; Hop-kins 2009).”
Monetary Policy Communications and Their Effects on Household Inflation Expectations (2018, neutral): “Moreover, de Quidt et al. (2018) show in settings similar to ours that these effects are plausibly small.”
Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents (2019, positive): “We note that recent experimental evidence has found little to no evidence of demand effects (De Quidt et al., 2018; Mummolo and Peterson, 2018; White et al., 2018), even when indicating the investigators’ preferred responses with heavy-handed messages.”
IQ, Expectations, and Choice (2019, neutral): “But we cannot fully disentangle whether an actual longer planning horizon drives this result or the fact that high-IQ agents assume that reporting longer planning horizons is a desirable answer—a classic issue of demand effects in experimental research (De Quidt et al. (2018)).”