Understanding and misunderstanding randomized controlled trials

By Angus Deaton, Nancy Cartwright. Social Science & Medicine (1967), 2016

1,545 citations recorded by the source corpus.

Randomized Controlled Trials (RCTs) are increasingly popular in the social sciences, not only in medicine. We argue that the lay public, and sometimes researchers, put too much trust in RCTs over other methods of investigation. Contrary to frequent claims in the applied literature, randomization does not equalize everything other than the treatment in the treatment and control groups, it does not automatically deliver a precise estimate of the average treatment effect (ATE), and it does not relieve us of the need to think about (observed or unobserved) covariates. Finding out whether an estimate was generated by chance is more difficult than commonly believed. At best, an RCT yields an unbiased estimate, but this property is of limited practical value. Even then, estimates apply only to the sample selected for the trial, often no more than a convenience sample, and justification is required to extend the results to other groups, including any population to which the trial sample belongs, or to any individual, including an individual in the trial. Demanding ‘external validity’ is unhelpful because it expects too much of an RCT while undervaluing its potential contribution. RCTs do indeed require minimal assumptions and can operate with little prior knowledge. This is an advantage when persuading distrustful audiences, but it is a disadvantage for cumulative scientific progress, where prior knowledge should be built upon, not discarded. RCTs can play a role in building scientific knowledge and useful predictions but they can only do so as part of a cumulative program, combining with other methods, including conceptual and theoretical development, to discover not ‘what works’, but ‘why things work’.

Subfields: Economics and Econometrics, General Health Professions, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/7b9130b6b0a22d5c69c43a95c698096c7f49abc1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In General Health Professions

In Statistics and Probability

Citations in the corpus

D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, background): “Indeed, observational data may, in spite of the three challenges above, provide improved statistical power and generalizability compared with experimental data [44].”
Use of Quasi-Experimental Research Designs in Education Research: Growth, Promise, and Challenges (2020, methodology): “With the proliferation of the use of QEDs across the social sciences, scholars are also raising concerns regarding the trade-offs inherent in the overreliance on the use of these methods for research and practice more broadly (Deaton & Cartwright, 2018; Pritchett, 2018).”
Protests as Strategic Games: Experimental Evidence from Hong Kong's Antiauthoritarian Movement* (2019, background): “Several recent articles have provided causal evidence on a bundle of “social” influences on protest participation: Enikolopov et al. (2016) present evidence that the diffusion of 1Emphasis added by the authors.”
Generalizing causal inferences from randomized trials: counterfactual and graphical identification (2019, background): “In economics, a particular type of interference is termed a “general equilibrium effect” [35].”
Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics (2020, result): “2It is worth noting that there have been thoughtful critiques of RCT as gold standard, see for example Deaton and Cartwright (2018). As far as the propensity for the published literature using a particular method to exhibit p-hacking and publication bias, our results indicate RCT outperforms the oth”
Universal Basic Income in the United States and Advanced Countries (2019, background): “No single program in the United States has a schedule like this, though Dube (2018) proposes one.”
What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results (2017, background): “In fact, there exists a lively debate over the relative merits of the aforementioned data types in forming the basis of more general inference (see, e.g., Levitt and List 2007, AlUbaydli and List 2015, Deaton and Cartwright 2016).”
Use of directed acyclic graphs (DAGs) to identify confounders in applied health research: review and recommendations (2020, background): “One approach is to conduct a randomized controlled experiment, but practical and ethical constraints mean this is only possible for a limited range of exposures.(2) Most causal effects must therefore be estimated from observational data; a notoriously difficult task that requires understanding, iden”
Women’s Empowerment and Economic Development: A Feminist Critique of Storytelling Practices in “Randomista” Economics (2020, background): “However, the usefulness of the empirical insights generated by randomista economics is limited by its failure to theorize change: “ causal conclusions require causal assumptions about how causality might work or fail to work or work in unexpected ways” (Deaton and Cartwright 2018b: 86.”