Estimation Considerations in Contextual Bandits

By Maria Dimakopoulou, Susan Athey, Guido W. Imbens. 2017

75 citations recorded by the source corpus.

Contextual bandit algorithms seek to learn a personalized treatment assignment policy, balancing exploration against exploitation. Although a number of algorithms have been proposed, there is little guidance available for applied researchers to select among various approaches. Motivated by the econometrics and statistics literatures on causal effects estimation, we study a new consideration to the exploration vs. exploitation framework, which is that the way exploration is conducted in the present may contribute to the bias and variance in the potential outcome model estimation in subsequent stages of learning. We leverage parametric and non-parametric statistical estimation methods and causal effect estimation methods in order to propose new contextual bandit designs. Through a variety of simulations, we show how alternative design choices impact the learning performance and provide insights on why we observe these effects.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Machine Learning Methods That Economists Should Know About (2019, background): “[2010], Dudik et al. [2011], Li et al. [2012], Dudik et al. [2014], Li et al. [2014], Swaminathan and Joachims [2015], Jiang and Li [2016], Thomas and Brunskill [2016], Kallus [2017]. Zhou et al. [2018] analyzes the case with more than two arms, extending the efficiency results of Athey and Wager [2”
Balanced Linear Contextual Bandits (2018, methodology): “We refer the reader to Appendix B of the supplemental material of the extended version of this paper (Dimakopoulou et al. 2017) for details on the datasets.”
The Impact of Machine Learning on Economics (2018, background): “Doudchenko and Imbens (2016) develop an approach inspired by synthetic controls (pioneered by Abadie et al.”
Economics, Fairness and Algorithmic Bias (2020, unknown): “Alternatively, researchers could use adaptive trial design (Barker et al., 2009; Palmer and Rosenberger, 1999) such as multi-armed bandits (Joseph et al., 2016; Jabbari et al., 2017; Dimakopoulou et al., 2017) to balance exploration and exploitation.”
Accurate Inference for Adaptive Linear Models (2017, methodology): “Finally, we note that the cost of increased variance can be mitigated by the use of information on the data collection policy, as demonstrated in [Nie et al., 2017, Dimakopoulou et al., 2017].”
Offline Multi-Action Policy Learning: Generalization and Optimization (2018, background): “For further discus- sion of the online learning literature, see, e.g., Li et al. (2010), Chu et al. (2011), Abbasi-Yadkori et al. (2011), Bubeck et al. (2012), Agrawal and Goyal (2013), Russo and Van Roy (2014), Li et al. (2017), Dimakopoulou et al. (2017)).”