Conformal inference of counterfactuals and individual treatment effects

By Lihua Lei, E.J. Candes. 2020

262 citations recorded by the source corpus.

Evaluating treatment effect heterogeneity widely informs treatment decision making. At the moment, much emphasis is placed on the estimation of the conditional average treatment effect via flexible machine learning algorithms. While these methods enjoy some theoretical appeal in terms of consistency and convergence rates, they generally perform poorly in terms of uncertainty quantification. This is troubling since assessing risk is crucial for reliable decision‐making in sensitive and uncertain environments. In this work, we propose a conformal inference‐based approach that can produce reliable interval estimates for counterfactuals and individual treatment effects under the potential outcome framework. For completely randomized or stratified randomized experiments with perfect compliance, the intervals have guaranteed average coverage in finite samples regardless of the unknown data generating mechanism. For randomized experiments with ignorable compliance and general observational studies obeying the strong ignorability assumption, the intervals satisfy a doubly robust property which states the following: the average coverage is approximately controlled if either the propensity score or the conditional quantiles of potential outcomes can be estimated accurately. Numerical studies on both synthetic and real data sets empirically demonstrate that existing methods suffer from a significant coverage deficit even in simple models. In contrast, our methods achieve the desired coverage with reasonably short intervals.

Subfields: Economics and Econometrics, Statistics and Probability, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Statistics and Probability

In Causal Inference & Discovery

Citations in the corpus

Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “Our work adds to several recent directions that explore CP in the context of risk-mitigating applications (Lei and Candès, 2020; Romano et al., 2020; Bates et al., 2020; Fisch et al., 2021a, inter alia ), or meta-learning settings (Fisch et al., 2021b).”
Conformal Risk Control (2022, methodology): “Then, under the 4.2 Q UANTILE RISK CONTROL Snell et al. (2022) generalizes Bates et al. (2021) to control the quantile of a monotone loss function conditional on ( X i , Y i ) ni =1 with probability 1 − δ over the calibration dataset for any user-specified tolerance parameter δ .”
Distribution-Free, Risk-Controlling Prediction Sets (2021, background): “Further extensions of conformal prediction address topics such as distribution estimation [25], causal inference [26], and handling or testing distribution shift [27–29].”
Conformal prediction beyond exchangeability (2022, methodology): “Since its publication, the ideas and techniques from Tibshirani et al. [2019] have been applied and extended in several ways.”