A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research

By M. Scriven. Journal of MultiDisciplinary Evaluation, 2008

187 citations recorded by the source corpus.

1 The causal wars are about what is to count as scientifically impeccable evidence of a causal connection, usually in the context of the evaluation of interventions into human affairs. The most recent battles are between those arguing that only the use of RCTs should be accepted as providing acceptable evidence (sometimes, the exotic regression discontinuity (RD) design is also allowed). The RCT or randomly controlled trial, is an experimental design involving at least two groups of subjects, the control group and the experimental group (a.k.a. study group, or treatment group), between which the subjects are distributed by a strictly random process (i.e., one with no exceptions), and which are not further identified or distinguished by any common factor besides the application of the experimental treatment to the experimental group. 2 The collateral damage comes from the policy that the RCT camp has been supporting with considerable success, here referred to as “the exclusionary policy,” which recommends that no (or almost no) programs be funded whose claims of good effects cannot be supported by RCT-based evidence. This means terminating many demonstrably excellent programs currently saving huge numbers of life-years. It is argued here that the exclusionary policy is not only based on false premises about the merits of (alleged) RCT designs, but even if its premises were true, it does not follow from them. This is an evaluation of a methodology, and such evaluations, common in all methodological literature and in the philosophy of science, are part of a sub-division of the discipline of evaluation sometimes referred to as intradisciplinary evaluation. It is an essential component of any discipline, since it is what justifies the term “discipline”—a field of study that could not distinguish reliably and validly between good and bad research methods could not be identified as a science rather than a pseudo-science. It should be noted in this introduction that the importance of the issue here lies not just in the key role of causation in evaluation, but also in the key role of causation in what might be called the middle ground of the body of investigative methods, those that lie between the highly theoretical and the highly localized techniques. This includes the key methods for many, perhaps most, practical investigations. It is not accidental that causation shares with evaluation the peculiar distinction of having been attacked as a concept that has no proper place in science. In each case, this was based on a superficial and elitist conception of science, the same elitist conception that argues for technology as an inferior discipline to—or as a mere spin-off from—“real science.” In all three cases, the truth of the matter is that the concept or practice under attack was a full-fledged and

Subfields: Management Science and Operations Research, Sociology and Political Science, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/607eb67224cb54d787fd0dab1e29dadd30723ff1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Sociology and Political Science

In Statistics and Probability

Citations in the corpus

Handbook on Impact Evaluation: Quantitative Methods and Practices (2009, background): “…of Development Effectiveness 389 (Duflo and Kremer 2003) as the Hawthorne and John Henry effects, these are not given the respect they warrant in most applications in developed countries (Scrivens 2008), let alone developing ones where the power of economists’ ideas arguably holds greater sway.”
The Importance of Qualitative Research for Causal Explanation in Education (2012, background): “For a particularly scathing critique of the idea that RCTs (randomized controlled trials) are the “gold standard” for causal inference, see Scriven (2008).”
How to do a good systematic review of effects in international development: a tool kit (2012, background): “Expectation effects may confound the causal mechanisms embodied in the particular intervention (Scriven 2008).”
Developing a Conceptual Framework for Life Skills Interventions Ψ (2013, background): “For example, regression discontinuity designs can be used to make causal inferences when random assignment is not practical or ethical (Lesik, 2006), while interrupted timeseries designs (Scriven, 2008), within-study comparisons (Cook et al.”
Adventure recreation and subjective well-being: a conceptual framework (2020, methodology): “…(Lesik, 2006), while interrupted time-series designs, withinstudy comparisons, success case method designs with a time-series element and qualitative impact on analyses offer alternative research design options (Coryn, Schroter, & Hanssen, 2009; Mohr, 1999; Scriven, 2008; Shadish et al., 2002).”
A Contribution to Current Debates in Impact Evaluation (2010, background): “But if they cannot be measured (so-called unobservables) and change over time, then randomization (randomized control trials, RCTs), and only randomization, can produce numerical estimates of impact which are free from bias – though, as critics correctly point out, even randomization may not do the ”
Systematic Review. What is the evidence of the impact of microfinance on the well-being of poor people? (2011, background): “However, RCTs in the social sciences generally do not assume non-interaction, hence the challenge to separate out the causal effects of a programme from all other factors that occur at the same time remains (Scriven 2008).”