Hindsight Credit Assignment

By Ani Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Gheshlaghi Azar, Bilal Piot, Nicolas Manfred Otto Heess, Hado van Hasselt, Greg Wayne, Satinder Singh, Doina Precup and 1 more. 2019

95 citations recorded by the source corpus.

We consider the problem of efficient credit assignment in reinforcement learning. In order to efficiently and meaningfully utilize new data, we propose to explicitly assign credit to past decisions based on the likelihood of them having led to the observed outcome. This approach uses new information in hindsight, rather than employing foresight. Somewhat surprisingly, we show that value functions can be rewritten through this lens, yielding a new family of algorithms. We study the properties of these algorithms, and empirically show that they successfully address important credit assignment challenges, through a set of illustrative tasks.

Subfields: Accounting, Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Accounting

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Many works have studied better credit assignment via state-association, learning an architecture which decomposes the reward function such that certain “important” states comprise most of the credit [50, 51, 12].”
Reward-Conditioned Policies (2019, methodology): “Like our method, Harutyunyan et al. [11] also learn the distribution of actions conditioned on future states or the trajectory return, but then utilize such models with standard RL techniques, such as policy gradients, to provide more effective credit assignment and variance reduction.”
Training Agents using Upside-Down Reinforcement Learning (2019, neutral): “Harutyunyan et al. [2019, defn.”
Inductive biases for deep learning of higher-level cognition (2020, background): “Future work should investigate how to apply these ideas in a reinforcement learning setting, in a manner which could also be related to the recent work on hindsight credit assignment (Harutyunyan et al., 2019).”
Policy Gradients Incorporating the Future (2021, background): “The need to bootstrap learning of the value function on its own estimates is known to lead to difficulties in practice, where one must achieve a careful balance between bias and variance [15, 49, 35, 23].”
Synthetic Returns for Long-Term Credit Assignment (2021, methodology): “Recent work has developed methods for more effective credit assignment using hindsight and counterfactual estimation (Harutyunyan et al., 2019; Mesnard et al., 2020; Buesing et al., 2018).”
Agent57: Outperforming the Atari Human Benchmark (2020, background): “…two types: ensuring that gradients correctly assign credit (Ke et al., 2017; Weber et al., 2019; Fortunato et al., 2019) and using values or targets to ensure correct credit is assigned (Arjona-Medina et al., 2019; Hung et al., 2019; Liu et al., 2019; Harutyunyan et al., 2019; Ferret et al., 2020).”
Counterfactual Credit Assignment in Model-Free Reinforcement Learning (2020, methodology): “…in reinforcement learning, point out the issue of non-identifiability of the correct SCM, and suggest a sufficient condition for identifiability; we discuss this issue in appendix G. Closely related to our work is Hindsight Credit Assignment, a concurrent approach from Harutyunyan et al. (2019).”