Counterfactual Credit Assignment in Model-Free Reinforcement Learning

By Thomas Mesnard, Theophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Ani Harutyunyan, Will Dabney, T. Stepleton, Nicolas Manfred Otto Heess, Arthur Guez and 3 more. 2020

73 citations recorded by the source corpus.

Credit assignment in reinforcement learning is the problem of measuring an action influence on future rewards. In particular, this requires separating skill from luck, ie. disentangling the effect of an action on rewards from that of external factors and subsequent actions. To achieve this, we adapt the notion of counterfactuals from causality theory to a model-free RL setup. The key idea is to condition value functions on future events, by learning to extract relevant information from a trajectory. We then propose to use these as future-conditional baselines and critics in policy gradient algorithms and we develop a valid, practical variant with provably lower variance, while achieving unbiasedness by constraining the hindsight information not to contain information about the agent actions. We demonstrate the efficacy and validity of our algorithm on a number of illustrative problems.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “To evaluate long-term credit assignment capabilities of our model, we consider a variant of the Key-to-Door environment proposed in Mesnard et al. [12].”
Policy Gradients Incorporating the Future (2021, methodology): “These methods were extended using a framework similar to ours, leveraging a backwards RNN to incorporate information in hindsight [22].”
Synthetic Returns for Long-Term Credit Assignment (2021, methodology): “Recent work has developed methods for more effective credit assignment using hindsight and counterfactual estimation (Harutyunyan et al., 2019; Mesnard et al., 2020; Buesing et al., 2018).”
On the link between conscious function and general intelligence in humans and machines (2022, background): “This approach provides the policy improvement and memory consolidation benefits of hippocampal replay, but has only extremely limited generalization beyond observed states and actions: technically, state-action value functions enable one-step interventions with novel actions (Mesnard et al., 2021).”