Synthetic Returns for Long-Term Credit Assignment

By David Raposo, Samuel Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matthew Botvinick, Hado van Hasselt, Francis Song. 2021

36 citations recorded by the source corpus.

Since the earliest days of reinforcement learning, the workhorse method for assigning credit to actions over time has been temporal-difference (TD) learning, which propagates credit backward timestep-by-timestep. This approach suffers when delays between actions and rewards are long and when intervening unrelated events contribute variance to long-term returns. We propose state-associative (SA) learning, where the agent learns associations between states and arbitrarily distant future rewards, then propagates credit directly between the two. In this work, we use SA-learning to model the contribution of past states to the current reward. With this model we can predict each state's contribution to the far future, a quantity we call"synthetic returns". TD-learning can then be applied to select actions that maximize these synthetic returns (SRs). We demonstrate the effectiveness of augmenting agents with SRs across a range of tasks on which TD-learning alone fails. We show that the learned SRs are interpretable: they spike for states that occur after critical actions are taken. Finally, we show that our IMPALA-based SR agent solves Atari Skiing -- a game with a lengthy reward delay that posed a major hurdle to deep-RL agents -- 25 times faster than the published state-of-the-art.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, neutral): “Similar to our long-term setting, some works have specifically shown such state-associative architectures can perform better in delayed reward settings (Arjona-Medina et al., 2018; Hung et al., 2019; Liu et al., 2019; Raposo et al., 2021).”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “However, the field has seen a number of alternative protocols used, including reporting the maximum evaluation score achieved during training [1, 3, 75] or across multiple runs [32, 47, 82].”