Causal Reasoning from Meta-reinforcement Learning

By Ishita Dasgupta, Jane X. Wang, Silvia Chiappa, Jovana Mitrovi'c, Pedro A. Ortega, David Raposo, Edward Hughes, Peter W. Battaglia, Matthew Botvinick, Zeb Kurth-Nelson. 2019

138 citations recorded by the source corpus.

Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether causal reasoning can emerge via meta-reinforcement learning. We train a recurrent network with model-free reinforcement learning to solve a range of problems that each contain causal structure. We find that the trained agent can perform causal reasoning in novel situations in order to obtain rewards. The agent can select informative interventions, draw causal inferences from observational data, and make counterfactual predictions. Although established formal causal reasoning algorithms also exist, in this paper we show that such reasoning can arise from model-free reinforcement learning, and suggest that causal reasoning in complex settings may benefit from the more end-to-end learning-based approaches presented here. This work also offers new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform -- and interpret -- experiments.

Subfields: Artificial Intelligence, Reinforcement Learning, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Causal Inference & Discovery

Citations in the corpus

Toward Causal Representation Learning (2021, methodology): “While Bengio et al. [25] and Ke et al. [131] focused on learning a causal model using neural networks with an unsupervised loss, the work of Dasgupta et al. [51] explores learning a causal model using an RL agent.”
Towards Causal Representation Learning (2021, methodology): “c) Counterfactuals: Counterfactual reasoning has been found to improve the data efficiency of RL algorithms [37, 165], improve performance [50], and it has been applied to communicate about past experiences in the multi-agent setting [68, 241].”
Causality for Machine Learning (2019, background): “btle (Lu et al., 2018; Gottesman et al., 2018). Recent work devoted to the ﬁeld between RL and causality includes (Bareinboim et al., 2015; Bengio et al., 2017; Lu et al., 2018; Buesing et al., 2018; Dasgupta et al., 2019; Zhang and Bareinboim, 2019). 9 Causal Representation Learning Traditional cau”
Learning Neural Causal Models from Unknown Interventions (2019, methodology): “Dasgupta et al. (2019) uses meta-learning to learn to make predictions under interventions.”
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms (2019, background): “More recently, Dasgupta et al. (2019) adopted a meta-learning approach to draw causal inferences from purely observational data.”
Inductive biases for deep learning of higher-level cognition (2020, neutral): “Preliminary work based on meta-learning (Ke et al., 2019; Dasgupta et al., 2019; Bengio et al., 2019) suggests that, parameterizing the correct variables and causal structures, the parameters of the graphical model capturing the (joint) observational distribution can be adapted faster to changes in…”
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning (2021, background): “As a result, there has been a recent surge of interest in learning causal models for deep reinforcement learning [de Haan et al., 2019, Dasgupta et al., 2019, Nair et al., 2019, Goyal et al., 2019, Goyal and Bengio, 2020, Rezende et al., 2020, Wang et al., 2021, Schölkopf et al., 2021].”
Visual Commonsense R-CNN (2020, background): “There has been a growing amount of efforts in marrying complementary strengths of deep learning and causal reasoning [49, 48] and have been explored in several contexts, including image classiﬁcation [8, 40], reinforcement learning [46, 11, 5] and adversarial learning [28, 26].”
Causal Induction from Visual Observations for Goal Directed Tasks (2019, unknown): “However, in contrast to Dasgupta et al. (2019) During training each episode samples one of K training environments and uses the interaction policy π I to probe the environment and collect a trajectory of visual observations.”
Meta-learning of Sequential Strategies (2019, background): “…have been designed accordingly, such as learning to identify a best option [Denil et al., 2016, Wang et al., 2016] or learning to reason causally [Dasgupta et al., 2019] In each case, the practitioner must ask the question: if the meta-learned strategy should have a capability X, what property Y…”
Tell me why! - Explanations support learning of relational and causal structure (2021, background): “Indeed, agents can meta-learn causal reasoning in simple settings (Dasgupta et al., 2019).”
Tell me why! Explanations support learning relational and causal structure (2021, background): “Indeed, agents can meta-learn causal reasoning in simple settings (Dasgupta et al., 2019) where causal variables are directly observable and actions allow strong interventions.”