An Assessment of Propensity Score Matching as a Nonexperimental Impact Estimator

By Juan-José Díaz, S. Handa. The Journal of human resources, 2005

222 citations recorded by the source corpus.

Not all policy questions can be addressed by social experiments. Nonexperimental evaluation methods provide an alternative to experimental designs but their results depend on untestable assumptions. This paper presents evidence on the reliability of propensity score matching (PSM), which estimates treatment effects under the assumption of selection on observables, using a social experiment designed to evaluate the PROGRESA program in Mexico. We find that PSM performs well for outcomes that are measured comparably across survey instruments and when a rich set of control variables is available. However, even small differences in the way outcomes are measured can lead to bias in the technique.

Subfields: Gender Studies, Safety Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/d725dce53ede52672b784c6d2b5a97ecd1337e59

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Gender Studies

1. Ayana N. Kee
2. P. Winters
3. A. Dillon
4. B. Davis
5. Vivian C. Wong
6. T. D. Cook
7. W. Shadish
8. Daniel W. Gingerich

In Safety Research

1. K. Covarrubias
2. Ayana N. Kee
3. P. Winters
4. A. Dillon
5. B. Davis
6. Vivian C. Wong
7. T. D. Cook
8. W. Shadish

In Statistics and Probability

1. W. Shadish
2. T. D. Cook
3. Ayana N. Kee
4. P. Winters
5. A. Dillon
6. Travis St.Clair
7. B. Davis
8. K. Hallberg

Citations in the corpus

• Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons (2008, methodology): “However, in the three instances analyzed here (Aiken et al., 1998; Bloom, Michalopoulos, & Hill, 2005; Diaz & Handa, 2006), such hidden bias either failed to operate or, less plausibly, was suppressed by selection bias of comparable magnitude operating in the opposite direction across all three…”
• The Experience of Conditional Cash Transfers in Latin America and the Caribbean (2006, unknown): “Diaz and Handa (2006) test whether PSM replicates the impact estimates of Progresa’s social experiment.”
• Understanding Off-the-Books Politics: Conducting Inference on the Determinants of Sensitive Behavior with Randomized Response Surveys (2010, background): “…those generated by randomized experiments have found that propensity score methods may approximate the experimental benchmark fairly well when the set of control variables is appropriately chosen and sufficiently rich (Heckman, Ichimura, and Todd 1997; Dehejia and Wahba 1999; Diaz and Handa 2006).”
• Propensity score matching and variations on the balancing test (2011, unknown): “…designed for the specific case of stratification on propensity scores is now often inappropriately used as a universal balancing test (for example, it was not appropriately used in Dehejia and Wahba (2002) or Diaz and Handa (2006) because matching methods other than stratification were used).”
• The Validity and Precision of the Comparative Interrupted Time-Series Design (2016, background): “…of many approaches to causal inference, including ITS (Fretheim et al., 2015), comparative regression discontinuity (Wing & Cook, 2013), matching (Diaz & Handa, 2006; Michalopoulos, Bloom, & Hill, 2004; Smith & Todd, 2005), and covariate measurement and selection (Cook & Steiner, 2010; Steiner,…”
• Do differences in the scale of irrigation projects generate different impacts on poverty and production (2011, background): “…of nonexperimental estimators against experimental benchmarks and against each other to verify whether identification assumptions can be plausibly asserted (see, for example, Heckman, Ichimura, and Todd 1997; Dehejia and Wahba 2002; Smith and Todd 2005; Dehejia 2005; Diaz and Handa 2006).”
• Feelings of Preparedness Among Alternatively Certified Teachers (2011, background): “Nevertheless, propensity score matching can strengthen causal inferences, particularly when there is detailed knowledge of the selection process (Diaz & Handa, 2006).”
• Selection Bias and Econometric Remedies in Accounting and Finance Research (2011, unknown): “12 weighting (where every observation in the untreated group is used with higher weights for closer observations and lower weights for more distant observations) (Diaz and Handa 2006). The PSM estimator for ATT is often defined as the mean outcome difference of treated and control firms matched by P”
• From protection to production: productive impacts of the Malawi Social Cash Transfer scheme (2012, background): “Some papers indicate that this loss of observations prevents the average treatment effect from being accurately estimated (Diaz and Handa 2006, Heinrich et al. 2010, Khandker et al. 2010, Gertler et al. 2011).”