Towards mental time travel: a hierarchical memory for reinforcement learning agents

By Andrew Kyle Lampinen, Stephanie C. Y. Chan, Andrea Banino, Felix Hill. 2021

61 citations recorded by the source corpus.

Reinforcement learning agents often forget details of the past, especially after delays or distractor tasks. Agents with common memory architectures struggle to recall and integrate across multiple timesteps of a past event, or even to recall the details of a single timestep that is followed by distractor tasks. To address these limitations, we propose a Hierarchical Chunk Attention Memory (HCAM), which helps agents to remember the past in detail. HCAM stores memories by dividing the past into chunks, and recalls by first performing high-level attention over coarse summaries of the chunks, and then performing detailed attention within only the most relevant chunks. An agent with HCAM can therefore"mentally time-travel"-- remember past events in detail without attending to all intervening events. We show that agents with HCAM substantially outperform agents with other memory architectures at tasks requiring long-term recall, retention, or reasoning over memory. These include recalling where an object is hidden in a 3D environment, rapidly learning to navigate efficiently in a new neighborhood, and rapidly learning and retaining new object names. Agents with HCAM can extrapolate to task sequences much longer than they were trained on, and can even generalize zero-shot from a meta-learning setting to maintaining knowledge across episodes. HCAM improves agent sample efficiency, generalization, and generality (by solving tasks that previously required specialized architectures). Our work is a step towards agents that can learn, interact, and adapt in complex and temporally-extended environments.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “Ideas from query (re)formulation [22, 59, 37, 51] may help agents expand the range of search exploration, and improved action exploration [40, 12, 49] and memory [53, 13, 23] mechanisms could help agents make better decisions over the long horizon and large action space.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “As a result, replay has become a very successful approach for tackling continual RL (Isele and Cosgun, 2018; Riemer et al., 2019; Rolnick et al., 2019; Oh et al., 2021; Henning et al., 2021; Liu et al., 2021b; Queeney et al., 2021; Chandak et al., 2021; Lampinen et al., 2021; Venuto et al., 2021; Li”
On the link between conscious function and general intelligence in humans and machines (2022, background): “Suﬃciently large buﬀers of experiences, which can be queried as needed during the learning of a novel task, provide powerful retrospective learning (Lampinen et al., 2021), but limited future-oriented imagination.”
Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learning (2022, unknown): “Adding explanation data (similar to self-QA) has been shown to improve few-shot performance via prompt manipulation in language models and for RL agents (Lampinen et al., 2021a; 2022), but has been less clearly connected to transfer learning.”