Why adaptively collected data have negative bias and how to correct for it

By Xinkun Nie, Xiaoying Tian, Jonathan E. Taylor, James Y. Zou. 2017

97 citations recorded by the source corpus.

From scientific experiments to online A/B testing, the previously observed data often affects how future experiments are performed, which in turn affects which data will be collected. Such adaptivity introduces complex correlations between the data and the collection procedure. In this paper, we prove that when the data collection procedure satisfies natural conditions, then sample means of the data have systematic \emph{negative} biases. As an example, consider an adaptive clinical trial where additional data points are more likely to be tested for treatments that show initial promise. Our surprising result implies that the average observed treatment effects would underestimate the true effects of each treatment. We quantitatively analyze the magnitude and behavior of this negative bias in a variety of settings. We also propose a novel debiasing algorithm based on selective inference techniques. In experiments, our method can effectively reduce bias and estimation error.

Subfields: Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Balanced Linear Contextual Bandits (2018, background): “Methods for reducing the bias due to adaptive data collection have also been studied for non-contextual multi-armed bandits (Villar, Bowden, and Wason 2015), (Nie et al. 2018), but the nature of the estimation in contextual ban-dits is qualitatively different.”
Estimation Considerations in Contextual Bandits (2017, background): “Methods for reducing the bias due to adaptive data collection have also been studied for non-contextual multi-armed bandits [63, 47], but the nature of the estimation in contextual bandits is qualitatively different.”