Generalization of Reinforcement Learners with Working and Episodic Memory

By Meire Fortunato, M. Tan, Steven Stenberg Hansen, Adrià Puigdomènech Badia, Gavin Buttimore, Charlie Deck, Joel Z. Leibo, Charles Blundell. 2019

74 citations recorded by the source corpus.

Memory is an important aspect of intelligence and plays a role in many deep reinforcement learning models. However, little progress has been made in understanding when specific memory systems help more than others and how well they generalize. The field also has yet to see a prevalent consistent and rigorous approach for evaluating agent performance on holdout data. In this paper, we aim to develop a comprehensive methodology to test different kinds of memory in an agent and assess how well the agent can apply what it learns in training to a holdout set that differs from the training set along dimensions that we suggest are relevant for evaluating memory-specific generalization. To that end, we first construct a diverse set of memory tasks that allow us to evaluate test-time generalization across multiple dimensions. Second, we develop and perform multiple ablations on an agent architecture that combines multiple memory systems, observe its baseline models, and investigate its performance against the task suite.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “Ideas from query (re)formulation [22, 59, 37, 51] may help agents expand the range of search exploration, and improved action exploration [40, 12, 49] and memory [53, 13, 23] mechanisms could help agents make better decisions over the long horizon and large action space.”
Towards mental time travel: a hierarchical memory for reinforcement learning agents (2021, background): “Most memories are based on LSTMs [20], often with additional key-value episodic memories [59, 14].”
Rapid Task-Solving in Novel Environments (2020, background): “Recent work has shown episodic memory to be effective in extending the capabilities of deep-RL agents to memory intensive tasks (Oh et al., 2016; Wayne et al., 2018; Ritter et al., 2018b; Fortunato et al., 2019).”
Do Transformers Need Deep Long-Range Memory? (2020, background): “, 2016) and recent memory-augmented agents for reinforcement learning, which utilise a distinct working memory with a single long-range episodic memory (Fortunato et al., 2019).”
Agent57: Outperforming the Atari Human Benchmark (2020, background): “…credit assignment can be categorized into roughly two types: ensuring that gradients correctly assign credit (Ke et al., 2017; Weber et al., 2019; Fortunato et al., 2019) and using values or targets to ensure correct credit is assigned (Arjona-Medina et al., 2019; Hung et al., 2019; Liu et al.,…”
Synthetic Returns for Long-Term Credit Assignment (2021, background): “This data structure has a long history in deep-RL, and is sometimes called “episodic memory” (Oh et al., 2016; Wayne et al., 2018; Ritter et al., 2018; Fortunato et al., 2019).”
Using Unity to Help Solve Intelligence (2020, background): “• Generalization of Reinforcement Learners with Working and Episodic Memory (Fortunato et al., 2019): A suite of thirteen diverse machine-learning tasks that require memory to solve, of which eight were developed with our approach.”