How Much Should We Trust Differences-in-Differences Estimates?
By Marianne Bertrand, Esther Duflo, Sendhil Mullainathan. 2004
10,994 citations recorded by the source corpus.
Most Difference-in-Difference (DD) papers rely on many years of data and focus on serially correlated outcomes. Yet almost all these papers ignore the bias in the estimated standard errors that serial correlation introduce4s. This is especially troubling because the independent variable of interest in DD estimation (e.g., the passage of law) is itself very serially correlated, which will exacerbate the bias in standard errors. To illustrate the severity of this issue, we randomly generate placebo laws in state-level data on female wages from the Current Population Survey. For each law, we use OLS to compute the DD estimate of its 'effect' as well as the standard error for this estimate. The standard errors are severely biased: with about 20 years of data, DD estimation finds an 'effect' significant at the 5% level of up to 45% of the placebo laws. Two very simple techniques can solve this problem for large sample sizes. The first technique consists in collapsing the data and ignoring the time-series variation altogether; the second technique is to estimate standard errors while allowing for an arbitrary covariance structure between time periods. We also suggest a third technique, based on randomization inference testing methods, which works well irrespective of sample size. This technique uses the empirical distribution of estimated effects for placebo laws to form the test distribution.
Subfields: Economics and Econometrics, Sociology and Political Science, Statistics and Probability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Economics and Econometrics
- James G. MacKinnon
- Morten Ørregaard Nielsen
- X. d'Haultfoeuille
- Guido W. Imbens
- M. D. Cattaneo
- C. Chaisemartin
- Esther Duflo
- Christian Hansen
In Sociology and Political Science
- M. D. Cattaneo
- C. Chaisemartin
- David H. Autor
- Guido W. Imbens
- X. d'Haultfoeuille
- Imran Rasul
- Pierre Cahuc
- Jacopo Ponticelli
In Statistics and Probability
- Guido W. Imbens
- Susan Athey
- M. D. Cattaneo
- Stefan Wager
- James G. MacKinnon
- Victor Chernozhukov
- Christian Hansen
- Alberto Abadie
Citations in the corpus
- Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program (2010, methodology): “For cases in which the number of available comparison regions is very small, one can use the longitudinal dimension of the data to produce placebo studies, as in Bertrand, Duflo, and Mullainathan (2004) where the dates of the placebo interventions are set at random.”
- The Impact of Length of the School Year on Student Performance and Earnings: Evidence from the German Short School Years (2007, neutral): “The block bootstrap and the unrestricted covariance matrix estimator implemented in Stata s cluster command did not perform well in the Bertrand et al. (2004) simulations for such a small number of states.”
- Responses to Entry in Multi-Sided Markets: The Impact of Craigslist on Local Newspapers (2014, background): “We cluster the error terms at the level of the newspaper to account for autocorrelation in the data within newspapers and over time (Bertrand et al. 2004).”
- Synth: An R Package for Synthetic Control Methods in Comparative Case Studies (2011, methodology): “Examples of placebo studies using the longitudinal dimension of the data are found in Appendix B of Abadie et al. (2010) and Bertrand, Duflo, and Mullainathan (2004).”
- Sharing riders: How bikesharing impacts bus ridership in New York City (2017, methodology): “Because of the potential for serial correlation across dates in this long time series, we use bootstrap standard errors (Bertrand et al., 2004).”
- Show Me the Way to Go Home: An Empirical Investigation of Ride-Sharing and Alcohol Related Motor Vehicle Fatalities (2017, background): “7Note also that diagnostics of the estimations suggest that the residuals do not suffer the serial correlation problems which often plague difference in difference estimations (Bertrand et al. 2004).”
- Which school attributes matter? The influence of school district performance and demographic composition on property values (2008, methodology): “Bertrand, Duflow, and Mullainathan [2] argue that standard errors in fixed effect models are biased by heteroscedasticity and autocorrelation, and Kezdi [29] demonstrates using simulations that clustered standard errors are suitable for use with autocorrelated and heteroscedastic panel data.”
- The Impact of Timing on Bidding Behavior in Procurement Auctions of Contracts with Private Costs (2012, positive): “We then test the difference across the three quantiles from the two models in Columns (1) and (2) in both Tables 5 and 6.”