Optimizing agent behavior over long time scales by transporting value

By Chia-Chun Hung, Timothy P. Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, Greg Wayne. 2018

139 citations recorded by the source corpus.

Humans prolifically engage in mental time travel. We dwell on past actions and experience satisfaction or regret. More than storytelling, these recollections change how we act in the future and endow us with a computationally important ability to link actions and consequences across spans of time, which helps address the problem of long-term credit assignment: the question of how to evaluate the utility of actions within a long-duration behavioral sequence. Existing approaches to credit assignment in AI cannot solve tasks with long delays between actions and consequences. Here, we introduce a paradigm where agents use recall of specific memories to credit past actions, allowing them to solve problems that are intractable for existing algorithms. This paradigm broadens the scope of problems that can be investigated in AI and offers a mechanistic account of behaviors that may inspire models in neuroscience, psychology, and behavioral economics. People are able to mentally time travel to distant memories and reflect on the consequences of those past events. Here, the authors show how a mechanism that connects learning from delayed rewards with memory retrieval can enable AI agents to discover links between past events to help decide better courses of action in the future.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Transformers can perform credit assignment directly via self-attention, in contrast to Bellman backups which slowly propagate rewards and are prone to “distractor” signals (Hung et al., 2019).”
Shaping Belief States with Generative Environment Models for RL (2019, methodology): “For all the types of RNNs (LSTM, LSTM+RMA, LSTM+Kanerva), we use the same network architecture for both the belief state and the simulation networks.”
Towards mental time travel: a hierarchical memory for reinforcement learning agents (2021, methodology): “Figure 2: We evaluated our model across a wide variety of task domains, some of which are modified or used from other recent memory papers [19, 21, 2, 48].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “Hung et al. (2018) introduce a method to better assign rewards that arrive significantly after a causative event.”
Hindsight Credit Assignment (2019, background): “The temporal value transport algorithm [11] also aims to propagate credit efficiently backward in time.”
Agent57: Outperforming the Atari Human Benchmark (2020, background): “…two types: ensuring that gradients correctly assign credit (Ke et al., 2017; Weber et al., 2019; Fortunato et al., 2019) and using values or targets to ensure correct credit is assigned (Arjona-Medina et al., 2019; Hung et al., 2019; Liu et al., 2019; Harutyunyan et al., 2019; Ferret et al., 2020).”
Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning (2019, background): “Temporal Value Transport (TVT) [16] relies on a memory reconstitutive module to retrieve past data for the visual RL problems, however, ours aims at decomposing the episodic return for general problems.”
Physically Embedded Planning Problems: New Challenges for Reinforcement Learning (2020, background): “These problems are notoriously diﬃcult for RL algorithms because of the wide variety of situations that can be encountered, the diﬃculty of ﬁnding successful plays by chance, and the long delay between actions and rewards (Arjona-Medina et al., 2019; Hung et al., 2019; Sutton and Barto, 2018).”
Self-Attentional Credit Assignment for Transfer in Reinforcement Learning (2019, unknown): “Existing backward-view credit assignment meth-ods [Arjona-Medina et al. , 2019; Hung et al. , 2018] require auxiliary terms to the loss function used to train agents, which can have detrimental effects to the learning process [de Bruin et al. , 2018], and rely on an external memory, which hinder…”
Counterfactual Credit Assignment in Model-Free Reinforcement Learning (2020, methodology): “Alternative networks could be used, such as attention-based networks (Hung et al., 2019) or RIMs (Goyal et al.”
Synthetic Returns for Long-Term Credit Assignment (2021, methodology): “We ran experiments on two other variants of the Key-to-Door task.”