For objective causal inference, design trumps analysis

By Donald B. Rubin. 2008

850 citations recorded by the source corpus.

For obtaining causal inferences that are objective, and therefore have the best chance of revealing scientific truths, carefully designed and executed randomized experiments are generally considered to be the gold standard. Observational studies, in contrast, are generally fraught with problems that compromise any claim for objectivity of the resulting causal inferences. The thesis here is that observational studies have to be carefully designed to approximate randomized experiments, in particular, without examining any final outcome data. Often a candidate data set will have to be rejected as inadequate because of lack of data on key covariates, or because of lack of overlap in the distributions of key covariates between treatment and control groups, often revealed by careful propensity score analyses. Sometimes the template for the approximating randomized experiment will have to be altered, and the use of principal stratification can be helpful in doing this. These issues are discussed and illustrated using the framework of potential outcomes to define causal effects, which greatly clarifies critical issues. 1. Randomized experiments versus observational studies. 1.1. Historical dichotomy between randomized and nonrandomized studies for causal effects. For may years, causal inference based on randomized experiments, as described, for example, in classic texts by Fisher (1935), Kempthorne (1952), Cochran and Cox (1950 )a ndCox (1958), was an entirely distinct endeavor than causal inference based on observational data sets, described, for example, in texts by Blalock (1964), Kenny (1979), Campbell and Stanley (1963), Cook and Campbell (1979), Rothman (1986), Lilienfeld and Lilienfeld (1976), Maddala (1977 )a ndCochran (1983). This began to change in the 1970’s when the use of potential outcomes, commonly used in the context of randomized experiments to define causal effects since Neyman (1923), was used to define causal effects in both randomized experiments and observational studies [Rubin (1974)]. This allowed the definition of assignment mechanisms [Rubin (1975)], with randomized experiments as special cases, thereby allowing

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Approximate residual balancing: debiased inference of average treatment effects in high dimensions (2016, background): “In contrast, stressing practical concerns, Rubin (2008) strongly argued that ‘designed-based’ inference leads to more credible conclusions in applications by better approximating randomized experiments.”
Leadership behavior and employee well-being: An integrated review and a future research agenda (2018, methodology): “Although the latter is an ideal method for establishing causation (for a more detailed discussion see Antonakis, Bendahan, Jacquart, & Lalive, 2014), it is often not viable for a variety of reasons (see Cook, Shadish, & Wong, 2008; Rubin, 2008).”
Covariate balancing propensity score by tailored loss functions (2016, methodology): “Later in the seminal work of Rosenbaum and Rubin (1983), the propensity score, defined as the conditional probability of receiving treatment given the covariates, was established as a fundamental tool to adjust for imbalance in more than just a few covariates.”
What You Can and Can’t Properly Do with Regression (2010, methodology): “This naturally leads to a variety of matching or post-stratification analysis methods that can be more robust than causal models (Rubin 2008).”
Causal Effects of Perceived Immutable Characteristics (2011, background): “(We distinguish the decider, who controls the outcome of interest, from the decision maker in Rubin, 2008, who may control treatment assignment.)”
Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research (2010, methodology): “Our study uses the standpoint of clearly separating “design” and “analysis” phases for causal inference, as recommended by Rubin and colleagues (Rubin, 2001; Stuart & Rubin, 2007; Rubin, 2008).”
Peer Influence and Delinquency (2019, background): “Given our reliance on observational data, the simple fact is that we will never be able to identify the true causal peer effect (Rubin 2008; see also Thomas et al. 2018).”
Using natural experiments to evaluate population health interventions: new Medical Research Council guidance (2012, background): “20 exposure (often referred to as the ‘ assignment process ’ even when there is no deliberate manipulation of individuals ’ exposure 21 ) is J Epidemiol Community Health 2012; 66 :1182 – 1186. doi:10.1136/jech-2011-200375”
Why Propensity Scores Should Not Be Used for Matching (2019, background): “‡Department of Political Science, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139; mit.edu/∼rnielsen, rnielsen@mit.edu, (857) 998-8039.”
Comment: The Design and Analysis of Gold Standard Randomized Experiments (2008, methodology): “The design and analysis of observational studies ideally follows the template of the design and analysis of randomized experiments, as I have strongly advocated recently (Rubin 2001, 2007, 2008).”
The Effect of Study Abroad on Intercultural Competence Among Undergraduate College Students (2013, methodology): “Furthermore, in addition to substantial problems in creating control groups demographically comparable to their treatment groups, neither study employed an analytical methodology designed to account for the selection effect when examining the effect of a given treatment in quasi-experimental studies”
Innovation intermediaries and collaboration: Knowledge–based practices and internal value creation (2018, neutral): “Seventh, as many theoretically justified control variables as possible were included to avoid the omitting of a regressor (Rubin, 2008; Shadish et al., 2002), which is recommended despite the cost of reduced efficiency (i.”