Causal Bandits: Learning Good Interventions via Causal Inference

By Finnian Lattimore, Tor Lattimore, Mark D. Reid. 2016

197 citations recorded by the source corpus.

We study the problem of using causal models to improve the rate at which good interventions can be learned online in a stochastic environment. Our formalism combines multi-arm bandits and causal inference to model a novel type of bandit feedback that is not exploited by existing approaches. We propose a new algorithm that exploits the causal feedback and prove a bound on its simple regret that is strictly better (in all quantities) than algorithms that do not use the additional causal information.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Causal Reasoning from Meta-reinforcement Learning (2019, background): “Despite advances in both (Or-tega and Stocker, 2015; Lattimore et al., 2016; Bramley et al., 2017; Forney et al., 2017; Sen et al., 2017; Parida et al., 2018), inducing models is expensive and typically requires simplifying assumptions.”
Balanced Linear Contextual Bandits (2018, background): “There is a number of recent works which approach contextual bandits through the framework of causality (Bareinboim, Forney, and Pearl 2015), (Bareinboim and Pearl 2015), (Forney, Pearl, and Bareinboim 2017), (Lattimore, Lattimore, and Reid 2016).”
Estimation Considerations in Contextual Bandits (2017, background): “There is a number of recent works which approach contextual bandits through the framework of causality [14, 15, 31, 41].”
A Survey of Learning Causality with Data (2018, methodology): “There is a line of work to improve RL models with causal knowledge as side information [87, 160].”
A Survey of Learning Causality with Data: Problems and Methods (2018, methodology): “The advantages of machine learning with causality include: (1) invariant prediction under environment changes [78, 101, 108], (2) model generality and interpretability [15, 127], and (3) performance improvement with theoretical guarantee [81, 144].”
Contextual Bandits with Latent Confounders: An NMF Approach (2016, background): “In [29], a causal model for observing feedback has been introduced in the best arm identification regime.”
A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning (2022, background): “Lee & Bareinboim (2018); Bareinboim et al. (2015); Lattimore et al. (2016); Moziﬁan et al. (2020); Volodin et al. (2020) also explore how to design an optimal intervention policy in bandits or RL settings.”
Identifying Best Interventions through Online Importance Sampling (2017, background): “We note that the authors in (Lattimore et al., 2016) discuss the possibility of a multi-phase approach, where clipper levels could change across phases.”