Causally Correct Partial Models for Reinforcement Learning

By Danilo Jimenez Rezende, Ivo Danihelka, George Papamakarios, Nan Rosemary Ke, Ray Jiang, Theophane Weber, Karol Gregor, Hamza Merzic, Fabio Viola, Jane X. Wang and 4 more. 2020

36 citations recorded by the source corpus.

In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent's next actions. However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images). For this reason, previous works have considered partial models, which model only part of the observation. In this paper, we show that partial models can be causally incorrect: they are confounded by the observations they don't model, and can therefore lead to incorrect planning. To address this, we introduce a general family of partial models that are provably causally correct, yet remain fast because they do not need to fully model future observations.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Shaking the foundations: delusions in sequence models for interaction and control (2021, background): “This problem 2 was ﬁrst pointed out in [14, 15], and related points were made in [19].”
Towards Causal Representation Learning (2021, methodology): “To address this issue, a model can use the backdoor criterion conditioning on its policy [200].”
Tell me why! Explanations support learning relational and causal structure (2021, background): “, Pearl, 2018), RL agents can intervene and thus can acquire causal knowledge (e.g., Dasgupta et al., 2019; Rezende et al., 2020).”
Planning from Pixels using Inverse Dynamics Models (2020, methodology): “9; end 3.5 D ATA C OLLECTION While training data need not be collected on-policy, we note that using an arbitrary data-collection policy may result in a causally-incorrect model (Rezende et al., 2020).”
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning (2021, background): “As a result, there has been a recent surge of interest in learning causal models for deep reinforcement learning [de Haan et al., 2019, Dasgupta et al., 2019, Nair et al., 2019, Goyal et al., 2019, Goyal and Bengio, 2020, Rezende et al., 2020, Wang et al., 2021, Schölkopf et al., 2021].”