Transfer Learning in Multi-Armed Bandits: A Causal Approach

By Junzhe Zhang, Elias Bareinboim. 2017

81 citations recorded by the source corpus.

Reinforcement learning (RL) agents have been deployed in complex environments where interactions are costly, and learning is usually slow. One prominent task in these settings is to reuse interactions performed by other agents to accelerate the learning process. Causal inference provides a family of methods to infer the effects of actions from a combination of data and qualitative assumptions about the underlying environment. Despite its success of transferring invariant knowledge across domains in the empirical sciences, causal inference has not been fully realized in the context of transfer learning in interactive domains. In this paper, we use causal inference as a basis to support a principled and more robust transfer of knowledge in RL settings. In particular, we tackle the problem of transferring knowledge across bandit agents in settings where causal effects cannot be identified by do-calculus [Pearl, 2000] and standard learning techniques. Our new identification strategy combines two steps – first, deriving bounds over the arms distribution based on structural knowledge; second, incorporating these bounds in a dynamic allocation procedure so as to guide the search towards more promising actions. We formally prove that our strategy dominates previously known algorithms and achieves orders of magnitude faster convergence rates than these algorithms. Finally, we perform simulations and empirically demonstrate that our strategy is consistently more efficient than the current (non-causal) state-of-the-art methods.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Causally Correct Partial Models for Reinforcement Learning (2020, background): “…interest in combining causal inference with RL research in the directions of non-model-based bandit algorithms (Bareinboim et al., 2015; Forney et al., 2017; Zhang & Bareinboim, 2017; Lee & Bareinboim, 2018; Bradtke & Barto, 1996; Lu et al., 2018) and causal discovery with RL (Zhu & Chen, 2019).”
Deconfounding Reinforcement Learning in Observational Settings (2018, background): “Zhang and Bareinboim (2017) leveraged causal inference to tackle the problem of transferring knowledge across bandit agents.”
A Survey of Learning Causality with Data: Problems and Methods (2018, background): “Other recent work bridging RL and causality includes causal approaches for transfer learning in RL models [142] and data-fusion for online reinforcement learners [40].”
Confounding-Robust Policy Improvement (2018, background): “Similar bounds studied in [33] in the transfer learning setting rely on no knowledge but the law of total probability.”
Transferable Contextual Bandit for Cross-Domain Recommendation (2018, background): “Though tUCB (Azar, Lazaric, and Brunskill 2013) and B-kl-UCB (Zhang and Bareinboim 2017) consider transfer learning for the bandit problem by transferring the estimated upper confidence bound, they differ from ours greatly. tUCB considers context-free multi-armed bandit problems which are inapplicab”
Counterfactual Data-Fusion for Online Reinforcement Learners (2017, background): “Recent work from causal analysis has addressed data-fusion for interventional quantities in reinforcement learning tasks (18).”