Heterogeneous Treatment Effects in Digital Experimentation

By Matt Taddy, Matt Gardner, Liyun Chen, Denise Draper. 2014

67 citations recorded by the source corpus.

Randomized controlled trials play an important role in how Internet companies predict the impact of policy decisions and product changes. In these `digital experiments', different units (people, devices, products) respond differently to the treatment. This article presents a fast and scalable Bayesian nonparametric analysis of such heterogeneous treatment effects and their measurement in relation to observable covariates. New results and algorithms are provided for quantifying the uncertainty associated with treatment effect measurement via both linear projections and nonlinear regression trees (CART and Random Forests). For linear projections, our inference strategy leads to results that are mostly in agreement with those from the frequentist literature. We find that linear regression adjustment of treatment effect averages (i.e., post-stratification) can provide some variance reduction, but that this reduction will be vanishingly small in the low-signal and large-sample setting of digital experiments. For regression trees, we provide uncertainty quantification for the machine learning algorithms that are commonly applied in tree-fitting. We argue that practitioners should look to ensembles of trees (forests) rather than individual trees in their analysis. The ideas are applied on and illustrated through an example experiment involving 21 million unique users of EBay.com.

Subfields: Management Science and Operations Research, Statistics and Probability, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

In Causal Inference & Discovery

Citations in the corpus

A Nonparametric Bayesian Analysis of Heterogenous Treatment Effects in Digital Experimentation (2014, methodology): “It applies beyond the two studied classes of HTE statistics, for example, an earlier draft of this work (Taddy et al. 2014) considered HTE summarization via moment conditions and our CART trees are just one possible prediction rule among many available machine learning tools.”
Estimating individual treatment effect: generalization bounds and algorithms (2016, background): “Recent work has also looked into how machine learning methods can help detect heterogeneous treatment effects when some data from randomized experiments is available (Taddy et al., 2016; Peysakhovich & Lada, 2016).”
Combining observational and experimental data to find heterogeneous treatment effects (2016, methodology): “The methods automatically find heterogeneous groups using a variety of tricks such as non-parametric procedures [10, 26], regularized regression [18], trees [14, 25], causal trees and forests [2, 27], deep neural networks [24], ‘virtual twin’ analyses [11] and mixtures of models [15].”