Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition

By Vincent Dorie, Jennifer L. Hill, Uri Shalit, Marc Scott, Daniel Cervone. 2017

350 citations recorded by the source corpus.

Statisticians have made great progress in creating methods that reduce our reliance on parametric assumptions. However this explosion in research has resulted in a breadth of inferential strategies that both create opportunities for more reliable inference as well as complicate the choices that an applied researcher has to make and defend. Relatedly, researchers advocating for new methods typically compare their method to at best 2 or 3 other causal inference strategies and test using simulations that may or may not be designed to equally tease out flaws in all the competing methods. The causal inference data analysis challenge, "Is Your SATT Where It's At?", launched as part of the 2016 Atlantic Causal Inference Conference, sought to make progress with respect to both of these issues. The researchers creating the data testing grounds were distinct from the researchers submitting methods whose efficacy would be evaluated. Results from 30 competitors across the two versions of the competition (black box algorithms and do-it-yourself analyses) are presented along with post-hoc analyses that reveal information about the characteristics of causal inference strategies and settings that affect performance. The most consistent conclusion was that methods that flexibly model the response surface perform better overall than methods that fail to do so. Finally new methods are proposed that combine features of several of the top-performing submitted methods.

Subfields: Artificial Intelligence, Statistics and Probability, Evaluation & Benchmarks, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Evaluation & Benchmarks

In Causal Inference & Discovery

Citations in the corpus

CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, background): “Even though causal inference is widely used in the life and social sciences, it has not had the same impact on machine learning and NLP in particular (Angrist and Pischke 2008; Dorie et al. 2019; Gentzel, Garant, and Jensen 2019).”
FLAME: A Fast Large-scale Almost Matching Exactly Approach to Causal Inference (2017, positive): “eling methods (that do not use matching) cannot. It helps determine what type of additional data must be collected to control for confounding. Interpretability is important: a recent empirical study (Dorie et al., 2019) suggests that CATEs estimated by manually calibrated methods can be better than ”
Quasi-oracle estimation of heterogeneous treatment effects (2017, background): “…several recent papers have proposed structural changes to various machine learning methods aimed at focusing on accurate estimation of τ ( · ) (Athey and Imbens, 2016; Hahn et al., 2017; Imai and Ratkovic, 2013; Powers et al., 2018; Shalit et al., 2017; Su et al., 2009; Wager and Athey, 2017).”
Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects (with Discussion) (2020, positive): “Based on an early draft of our manuscript Dorie et al. (2019) also evaluated a version of BART that included an estimate of the propensity score, which was one of the top methods on bias and RMSE for estimating the sample average treatment effect among the treated (SATT).”
A Survey on Causal Inference (2020, methodology): “treated children with non-white moms. The whole datasets can be downloaded from https://drive.google.com/file/d/0B7pG5PPgj6A3N09ibmFwNWE1djA/view, and the summarization of this year’s challenge is in [36]. 2017 Challenge: ACIC 2017 challenge focused on the estimation and inference for conditional av”
Bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effects (2017, neutral): “Gelman, A. et al. (2006). Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayesian Analysis 1 515–534. Giles, D. and Rayner, A. (1979). The mean squared errors of the maximum likelihood and natural-conjugate Bayes regression estimators. J”
Atlantic Causal Inference Conference (ACIC) Data Analysis Challenge 2017 (2019, methodology): “These covariates were also used in the 2016 ACIC Data Analysis Challenge [Dorie et al., 2019].”
CAUSAL INTERPRETATIONS OF BLACK-BOX MODELS (2019, neutral): “The Atlantic Causal Inference Conference has even held a machine-learning-style competition to compare the various causal inference methods every year since 2016 (Dorie et al. 2019).”
The Secrets of Machine Learning: Ten Things You Wish You Had Known Earlier to be More Effective at Data Analysis (2019, methodology): “ct the treatment of another). One of the current methods to compute treatment eects is using the machine learning method BART (Bayesian Additive Regression Trees) due to its high quality performance [15,57]. BART models are not interpretable. For interpretability, matching methods are a good choice,”
Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates (2020, methodology): “Semi-synthetic datasets use real covariates and synthetically generate treatment and outcome, as in the 2016 Atlantic Causal Inference Competition (Dorie et al., 2019).”