Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

By Lars Buesing, Theophane Weber, Yori Zwols, Sébastien Racanière, Arthur Guez, Jean-Baptiste Lespiau, Nicolas Manfred Otto Heess. 2018

167 citations recorded by the source corpus.

Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for model-based policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on Importance Sampling which re-weight data, CF-GPS leverages a model to explicitly consider alternative outcomes, allowing the algorithm to make better use of experience data. We find empirically that these advantages translate into improved policy evaluation and search results on a non-trivial grid-world task. Finally, we show that CF-GPS generalizes the previously proposed Guided Policy Search and that reparameterization-based algorithms such Stochastic Value Gradient can be interpreted as counterfactual methods.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Causal Inference & Discovery

Citations in the corpus

Model-Based Reinforcement Learning for Atari (2019, background): “, 2015), creating intrinsic motivation (Schmidhuber, 2010) or counterfactual reasoning (Buesing et al., 2019). One of the exciting benefits of model-based learning is the promise to substantially improve sample efficiency of deep reinforcement learning (see Chapter 8 in Sutton & Barto (2017)). Our w”
Toward Causal Representation Learning (2021, background): “3) Counterfactuals: Counterfactual reasoning has been found to improve the data efﬁciency of RL algorithms [38], [164] and improve performance [51], and it has been applied to communicate about past experiences in the multiagent setting [69], [241].”
Towards Causal Representation Learning (2021, background): “c) Counterfactuals: Counterfactual reasoning has been found to improve the data efficiency of RL algorithms [37, 165], improve performance [50], and it has been applied to communicate about past experiences in the multi-agent setting [68, 241].”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, methodology): “Recently, an algorithm based on probabilistic counterfactually-guided policy search has been proposed in the POMDP setting [7].”
Causally Correct Partial Models for Reinforcement Learning (2020, background): “…explored for off-policy policy evaluation (Sutton & Barto, 2018; Precup, 2000; Precup et al., 2001; Munos et al., 2016; Espeholt et al., 2018; Nachum et al., 2018) as well as for counter-factual policy evaluation (Buesing et al., 2019) and for counter-factual model evaluation (Bottou et al., 2013).”
Causality for Machine Learning (2019, background): “Recent work devoted to the field between RL and causality includes (Bareinboim et al., 2015; Bengio et al., 2017; Lu et al., 2018; Buesing et al., 2018; Dasgupta et al., 2019; Zhang and Bareinboim, 2019).”
Deconfounding Reinforcement Learning in Observational Settings (2018, background): “To the best of our knowledge, however, little work has been done in this promising area of integrating RL with causal inference (Bareinboim et al., 2015; Forney et al., 2017; Buesing et al., 2019).”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “Recently there has been some work using neural networks to approximate counterfactual models (Buesing et al., 2019; Johansson et al., 2016).”
Causal Induction from Visual Observations for Goal Directed Tasks (2019, unknown): “Recent work has shown that causal learning can take advantage of the representational power of deep learning methods for inducing causal relationships from interventions (Dasgupta et al., 2019) and for improving policy learning via counterfactual reasoning (Buesing et al., 2018).”
Hindsight Credit Assignment (2019, background): “Buesing et al. [2] apply this idea to POMDPs, where it can be viewed as reasoning about events in hindsight.”
Options as responses: Grounding behavioural hierarchies in multi-agent RL (2019, unknown): “This is similar to hindsight policy gradient [28] and experience replay [2] and other counterfactual RL works [6, 9, 21].”
Learning to Predict Without Looking Ahead: World Models Without Forward Prediction (2019, background): “Much of the motivation of model-based reinforcement learning (RL) derives from the potential utility of learned models for downstream tasks, like prediction [13, 15], planning [1, 34, 39, 40, 42, 63], and counterfactual reasoning [9, 27].”
Do recent advancements in model-based deep reinforcement learning really improve data efficiency? (2019, background): “Ability to learn the model of the environment was subsequently leveraged to successfully improve different aspects of deep RL (Racani`ere et al., 2017; Oh et al., 2017; Buesing et al., 2018; Ha & Schmidhuber, 2018).”
Synthetic Returns for Long-Term Credit Assignment (2021, methodology): “Recent work has developed methods for more effective credit assignment using hindsight and counterfactual estimation (Harutyunyan et al., 2019; Mesnard et al., 2020; Buesing et al., 2018).”
MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming (2019, background): “, 2017), and reinforcement learning (Foerster et al., 2018; Forney et al., 2017; Buesing et al., 2019).”
Counterfactual Credit Assignment in Model-Free Reinforcement Learning (2020, methodology): “To this end, we assume that the MDP (X ,A, p, r, γ) in question is generated by an underlying structural causal models (SCM) analogous to (Buesing et al., 2019; Zhang, 2020).”
Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models (2019, methodology): “…of δo). δo := Eπ̂[R(τ)|O1 = o]− Eobs[R(τ)|O1 = o] = ∫ τ pπobs(τ |O1 = o1)Eu∼pπobs (u|τ)[R(τπ̂(u))−R(τ)] The proofs of Lemma 1 and Corollary 1 are very similar to (and are essentially implied by) Lemma 1 from Buesing et al. (2019), but are given in self-contained form in the supplement.”
On the link between conscious function and general intelligence in humans and machines (2022, background): “Again, incorporating further structure into the model can aid generalization, by for example disentangling dynamic and static components (Kim et al., 2020), or learning causal models (Buesing et al., 2018; Gasse et al., 2021).”