The use of propensity scores to assess the generalizability of results from randomized trials

By Elizabeth A. Stuart, Stephen R. Cole, Catherine P. Bradshaw, P. Leaf. 2011

550 citations recorded by the source corpus.

Summary. Randomized trials remain the most accepted design for estimating the effects of interventions, but they do not necessarily answer a question of primary interest: will the programme be effective in a target population in which it may be implemented? In other words, are the results generalizable? There has been very little statistical research on how to assess the generalizability, or ‘external validity’, of randomized trials. We propose the use of propensity‐score‐based metrics to quantify the similarity of the participants in a randomized trial and a target population. In this setting the propensity score model predicts participation in the randomized trial, given a set of covariates. The resulting propensity scores are used first to quantify the difference between the trial participants and the target population, and then to match, subclassify or weight the control group outcomes to the population, assessing how well the propensity‐score‐adjusted outcomes track the outcomes that are actually observed in the population. These metrics can serve as a first step in assessing the generalizability of results from randomized trials to target populations. The paper lays out these ideas, discusses the assumptions underlying the approach and illustrates the metrics by using data on the evaluation of a schoolwide prevention programme called ‘Positive behavioral interventions and supports’.

Subfields: Developmental and Educational Psychology, General Health Professions, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Developmental and Educational Psychology

In General Health Professions

In Statistics and Probability

Citations in the corpus

From Local to Global: External Validity in a Fertility Natural Experiment (2015, unknown): “This also raises concerns about generating extrapolations solely on the basis of micro-level data, an issue that Hotz, Imbens, and Mortimer (2005), Stuart et al. (2011), and Hartman et al. (2015) were unable to investigate due to the limitations of their evidence bases.”
Mechanism Experiments and Policy Evaluations (2011, background): “But meta-analyses of black-box clinical trials showed that statins improved health outcomes even among people who already had relatively low levels of blood cholesterol at baseline (Golomb, Criqui, White, and Dimsdale, 2004; Wilt et al., 2004; Thavendiranathan et al., 2006).”
Worth Weighting? How to Think About and Use Weights in Survey Experiments (2017, background): “Tipton (2013) investigates the associated bias-variance trade-offs due to stratification, and gives advice as to when stratification will be effective.”
A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results (2015, background): “The resulting weights can be used to understand differences between the trial and target realworld populations, and to “project” the RCT efficacy to the target population, in effect providing an estimate of the efficacy that would be observed were the trial to be conducted in a more representative e”
Improving Generalizations From Experiments Using Propensity Score Subclassification (2013, methodology): “Improving Generalizations From Experiments The current article extends the work of Stuart et al. (2011) and Hedges and O’Muircheartaigh (2011) in three ways.”
Inference for Nonprobability Samples (2017, background): “Stuart et al. (2011) considers the use of propensity cores to generalize results from randomized trials to populations. Kaizar (2015) reviews approaches that have been proposed for combining randomized and nonrandomized studies in the estimation of treatment efficacy.”
Get real in individual participant data (IPD) meta‐analysis: a review of the methodology (2015, background): “Most trials are conducted to establish efficacy and safety of a single treatment in a specific study setting, and therefore it is often difficult to translate this evidence to “real-world” effectiveness where different concerns may be at play (Prevost et al., 2000; Stuart et al., 2011).”
How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations (2014, methodology): “Stuart et al (2011) also propose another method for assessing generalizability that requires knowledge of Y(0) in both the experiment and population.”
Stable Weights that Balance Covariates for Estimation With Incomplete Outcome Data (2015, methodology): “Also, this method can be extended to obtain representativeness of randomized experiments and matched observational studies (Stuart et al. 2011).”