Combining observational and experimental data to find heterogeneous treatment effects

By Alex Peysakhovich, Akos Lada. 2016

37 citations recorded by the source corpus.

Every design choice will have different effects on different units. However traditional A/B tests are often underpowered to identify these heterogeneous effects. This is especially true when the set of unit-level attributes is high-dimensional and our priors are weak about which particular covariates are important. However, there are often observational data sets available that are orders of magnitude larger. We propose a method to combine these two data sources to estimate heterogeneous treatment effects. First, we use observational time series data to estimate a mapping from covariates to unit-level effects. These estimates are likely biased but under some conditions the bias preserves unit-level relative rank orderings. If these conditions hold, we only need sufficient experimental data to identify a monotonic, one-dimensional transformation from observationally predicted treatment effects to real treatment effects. This reduces power demands greatly and makes the detection of heterogeneous effects much easier. As an application, we show how our method can be used to improve Facebook page recommendations.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Causal Effect Inference with Deep Latent-Variable Models (2017, neutral): “With the availability of large datasets in domains such as healthcare and education, there is much interest in developing methods for learning individual-level causal effects from observational data [42, 53, 25, 43].”
The Impact of Machine Learning on Economics (2018, background): “Beyond bandits, other themes include combining experimental and observational data to improve precison of estimates (see, e.g., Peysakhovich and Lada (2016)), and making use of large numbers of related experiments when drawing conclusions.”
Estimating individual treatment effect: generalization bounds and algorithms (2016, background): “Recent work has also looked into how machine learning method can help detect heterogeneous treatment effects when some data from randomized experiments is available (Taddy et al., 2016; Peysakhovich & Lada, 2016).”