Unsupervised Predictive Memory in a Goal-Directed Agent

By Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska-Barwinska, Jack W. Rae, Piotr Wojciech Mirowski, Joel Z. Leibo, Adam Santoro and 14 more. 2018

206 citations recorded by the source corpus.

Animals execute goal-directed behaviours despite the limited range and scope of their sensors. To cope, they explore environments and store memories maintaining estimates of important information that is not presently available. Recently, progress has been made with artificial intelligence (AI) agents that learn to perform tasks from sensory input, even at a human level, by merging reinforcement learning (RL) algorithms with deep neural networks, and the excitement surrounding these results has led to the pursuit of related ideas as explanations of non-human animal learning. However, we demonstrate that contemporary RL algorithms struggle to solve simple tasks when enough information is concealed from the sensors of the agent, a property called "partial observability". An obvious requirement for handling partially observed tasks is access to extensive memory, but we show memory is not enough; it is critical that the right information be stored in the right format. We develop a model, the Memory, RL, and Inference Network (MERLIN), in which memory formation is guided by a process of predictive modeling. MERLIN facilitates the solution of tasks in 3D virtual reality environments for which partial observability is severe and memories must be maintained over long durations. Our model demonstrates a single learning agent architecture that can solve canonical behavioural tasks in psychology and neurobiology without strong simplifying assumptions about the dimensionality of sensory input or the duration of experiences.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Reinforcement Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, methodology): “, 2019)and MERLIN (Wayne et al., 2018) as well the expressive generative model in (Gregor et al.”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “Ideas from query (re)formulation [22, 59, 37, 51] may help agents expand the range of search exploration, and improved action exploration [40, 12, 49] and memory [53, 13, 23] mechanisms could help agents make better decisions over the long horizon and large action space.”
Learning Latent Dynamics for Planning from Pixels (2018, background): “, 2018), combining the benefits of model-based and model-free approaches (Kalweit & Boedecker, 2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha & Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 2018), and pure video predictio”
Stabilizing Transformers for Reinforcement Learning (2019, methodology): “Despite progress at designing more expressive memory architectures (Graves et al., 2016; Wayne et al., 2018; Santoro et al., 2018) that perform better than LSTMs in memory-based tasks and partially-observable environments, they have not seen widespread adoption in RL agents perhaps due to their…”
Neural Predictive Belief Representations (2018, background): “Wayne et al. (2018) showed that the representation of their proposed agent architecture is able to capture the absolute position of the goal in a large-maze navigation task.”
Towards mental time travel: a hierarchical memory for reinforcement learning agents (2021, background): “Furthermore, both HCAM and EPN substantially outperform Merlin [60], a strong baseline that learns to store vector memories, rather than the rich representations of sequences stored by HCAM (or transitions stored by EPN).”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, neutral): “…(Das et al., 2018; Gordon et al., 2018), and navigation (Matuszek et al., 2013; Hemachandra et al., 2015; Duvallet et al., 2016; Zhu et al., 2017; de Vries et al., 2018; Yuke Zhu, 2017; Mousavian et al., 2018; Wayne et al., 2018; Wang et al., 2018a; Mirowski et al., 2017; 2018; Zamir et al., 2018).”
Fast deep reinforcement learning using online adjustments from the past (2018, background): “There has been a lot of recent work on using memory-augmented neural networks as a function approximation for RL agents: using LSTMs [Bakker et al., 2003, Hausknecht and Stone, 2015], or more sophisticated architectures [Graves et al., 2016, Oh et al., 2016, Wayne et al., 2018].”
S-TRIGGER: Continual State Representation Learning via Self-Triggered Generative Replay (2019, unknown): “They have recently shown promising results as State Representation models for RL [7], [19].”
Synthetic Returns for Long-Term Credit Assignment (2021, background): “This data structure has a long history in deep-RL, and is sometimes called “episodic memory” (Oh et al., 2016; Wayne et al., 2018; Ritter et al., 2018; Fortunato et al., 2019).”
Unsupervised Learning of Sensorimotor Affordances by Stochastic Future Prediction (2018, background): “Other methods have shown that having access to the representation learned by an unsupervised prediction model improves the performance of reinforcement learning agents on a broad range of tasks [16, 50].”
Reinforcement Learning, Fast and Slow. (2019, unknown): “A full description of the detailed ‘wiring’ of this RL agent is beyond the scope of the present paper (but can be found in reference [51]).”
Generalization of Reinforcement Learners with Working and Episodic Memory (2019, methodology): “Reconstruction losses have also been used as an auxiliary task (Jaderberg et al., 2016; Wayne et al., 2018) and we include this as a baseline in our experiments.”
Learning to plan with uncertain topological maps (2020, unknown): “Extensions to memory based neural approaches began with Neural Turing Machines [19] and Diﬀerentiable Neural Computers [20], and have since been adapted to expand the capacity of Deep RL agents [56].”
Mastering Atari with Discrete World Models (2020, background): “, 2018; Gregor and Besse, 2018), or combine the benefits of model-based and model-free approaches (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 20”
If deep learning is the answer, what is the question? (2020, background): “However, to date, wiring these generative models up with control systems to build intelligent agents has proved challenging, despite some promising efforts 78 .”
Continual State Representation Learning for Reinforcement Learning using Generative Replay (2018, unknown): “They have recently shown promising results as State Representation models for RL (Ha and Schmidhuber, 2018; Wayne et al., 2018).”
Learning Deployable Navigation Policies at Kilometer Scale from a Single Traversal (2018, background): “Models can be built from the real environment in which the agent is embodied in order to ﬁt directly the speciﬁcs of the robot [20, 21, 22, 23].”
Hierarchical visuomotor control of humanoids (2018, background): “As techniques have matured, it is now possible to train high-dimensional vision-based policies from scratch to generate a range of interesting behaviors ranging from game-playing to navigation (Jaderberg et al., 2018; OpenAI, 2018; Wayne et al., 2018).”