Deconfounding Reinforcement Learning in Observational Settings

By Chaochao Lu, Bernhard Schölkopf, José Miguel Hernández-Lobato. 2018

79 citations recorded by the source corpus.

We propose a general formulation for addressing reinforcement learning (RL) problems in settings with observational data. That is, we consider the problem of learning good policies solely from historical data in which unobserved factors (confounders) affect both observed actions and rewards. Our formulation allows us to extend a representative RL algorithm, the Actor-Critic method, to its deconfounding variant, with the methodology for this extension being easily applied to other RL algorithms. In addition to this, we develop a new benchmark for evaluating deconfounding RL algorithms by modifying the OpenAI Gym environments and the MNIST dataset. Using this benchmark, we demonstrate that the proposed algorithms are superior to traditional RL methods in confounded environments with observational data. To the best of our knowledge, this is the first time that confounders are taken into consideration for addressing full RL problems with observational data. Code is available at this https URL.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability, Reinforcement Learning, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yoshua Bengio
2. Sergey Levine
3. Dacheng Tao
4. Bernhard Schölkopf
5. Ruslan Salakhutdinov
6. Pieter Abbeel
7. Quoc V. Le
8. Kyunghyun Cho

In Management Science and Operations Research

1. Bernhard Schölkopf
2. Mingming Gong
3. Yoshua Bengio
4. Dacheng Tao
5. Frederic Besse
6. Theophane Weber
7. Karol Gregor
8. Nan Rosemary Ke

In Statistics and Probability

1. Anirudh Goyal
2. Stefan Bauer
3. Yoshua Bengio
4. Francesco Locatello
5. Mingming Gong
6. Nan Rosemary Ke
7. Nal Kalchbrenner
8. Dacheng Tao

In Reinforcement Learning

1. Sergey Levine
2. Yoshua Bengio
3. Anirudh Goyal
4. Pieter Abbeel
5. Bernhard Schölkopf
6. Karol Gregor
7. Danilo Jimenez Rezende
8. Dacheng Tao

In Causal Inference & Discovery

1. Yoshua Bengio
2. Anirudh Goyal
3. Stefan Bauer
4. Francesco Locatello
5. Nan Rosemary Ke
6. Nal Kalchbrenner
7. Bernhard Schölkopf
8. Jovana Mitrovi'c

Citations in the corpus

• Toward Causal Representation Learning (2021, background): “However, as soon as off-policy learning is considered, in particular, in the batch (or observational) setting [146], issues of causality become subtle [82], [165].”
• Towards Causal Representation Learning (2021, background): “An emerging line of work devoted to the intersection of RL and causality includes [13, 21, 164, 37, 50, 275, 1].”
• Causally Correct Partial Models for Reinforcement Learning (2020, background): “Prior to this work, there has been a growing interest in combining causal inference with RL research in the directions of nonmodel-based bandit algorithms (Bareinboim et al., 2015; Forney et al., 2017; Zhang & Bareinboim, 2017; Lee & Bareinboim, 2018; Bradtke & Barto, 1996; Lu et al., 2018) and caus”
• Causality for Machine Learning (2019, methodology): “One way to do this is to employ model-based RL using SCMs, an approach which can help address a problem of confounding in RL where time-varying and time-invariant unobserved confounders influence both actions and rewards (Lu et al., 2018).”
• A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning (2022, background): “However, solving real-world problems with RL is still a challenging problem because the sample efficiency of RL is low while the data in many applications is limited or expensive to obtain (Gottesman et al., 2018; Lu et al., 2018; 2020; Kiran et al., 2020).”