Neural Map: Structured Memory for Deep Reinforcement Learning

By Emilio Parisotto, Ruslan Salakhutdinov. 2017

286 citations recorded by the source corpus.

A critical component to enabling intelligent reasoning in partially observable environments is memory. Despite this importance, Deep Reinforcement Learning (DRL) agents have so far used relatively simple memory architectures, with the main methods to overcome partial observability being either a temporal convolution over the past k frames or an LSTM layer. More recent work (Oh et al., 2016) has went beyond these architectures by using memory networks which can allow more sophisticated addressing schemes over the past k frames. But even these architectures are unsatisfactory due to the reason that they are limited to only remembering information from the last k frames. In this paper, we develop a memory system with an adaptable write operator that is customized to the sorts of 3D environments that DRL agents typically interact with. This architecture, called the Neural Map, uses a spatially structured 2D memory image to learn to store arbitrary information about the environment over long time lags. We demonstrate empirically that the Neural Map surpasses previous DRL memories on a set of challenging 2D and 3D maze environments and show that it is capable of generalizing to environments that were not seen during training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Value Prediction Network (2017, background): “A lot of variations of DQN have been proposed for learning better state representation [37, 16, 9, 22, 36, 24], including the use of memory-based networks for handling partial observability [9, 22, 24], estimating both state-values and advantage-values as a decomposition of Q-values [37], learning s”
Shaping Belief States with Generative Environment Models for RL (2019, background): “Extracting a consistent high-level representation of the environment such as the bird’s eye view "map" from merely first-person views and actions in a completely unsupervised manner is a notoriously difficult task [12] and a lot of the success in addressing this problem is due to the injection of a ”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Zhang et al. (2017b) proposed Neural SLAM , based on the Neural Map proposed by Parisotto and Salakhutdinov (2017), where the Neural Turing Machine (NTM) is deployed for the DRL agent to interact with.”
Cognitive Mapping and Planning for Visual Navigation (2017, methodology): “[10, 27, 45, 61] use similar 2D spatial representations, Mirowski et al.”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “Parisotto & Salakhutdinov (2017) propose a new architecture which stores information of the environment on a 2D map.”
TextWorld: A Learning Environment for Text-based Games (2018, methodology): “This benchmark is inspired by the task proposed in Parisotto and Salakhutdinov [2017], where the agent spawns in a randomly generated maze and must ﬁnd a speciﬁc object.”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “More recent work, where the memory was given a 2D structure in order to resemble a spatial map, hints at future research where more specialised memory structures will be developed to address specific problems, such as 2D or 3D navigation [84].”
Stabilizing Transformers for Reinforcement Learning (2019, methodology): “Priors are induced through the design of the speciﬁc read/write operations, such as those resembling a digital computer (Wayne et al., 2018; Graves et al., 2016) or an environment map (Parisotto & Salakhutdinov, 2018; Gupta et al., 2017).”
On Evaluation of Embodied Navigation Agents (2018, background): “The next level of architectural sophistication is to use more advanced memory mechanisms that support the construction of rich internal representations of the agent’s environment [12, 13, 22, 23, 27].”
Semi-parametric Topological Memory for Navigation (2018, methodology): “…et al. (2017) that uses ground-truth depth maps and ego-motion, the method of Gupta et al. (2017) that operates on a discrete grid given ground-truth ego-motion, and the approach of Parisotto & Salakhutdinov (2018) that requires the knowledge of ground-truth global coordinates of the agent.”
iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, background): “In these environments, agents learn to engage in physical interactions [1, 2], navigate based on sensor signals [3, 4, 5, 6, 7], or plan long-horizon tasks [8, 9, 10, 11].”
Simultaneous Mapping and Target Driven Navigation (2019, methodology): “The methods which explicitly learn a spatial representation of the environment, proposed task-dependent differentiable spatial memories to represent the environment [9, 10, 4, 8, 17, 28] and typically focus on goal or target driven navigation, localization or exploration tasks.”
Learning Exploration Policies for Navigation (2019, methodology): “Some of these design choices were inspired by recent work from Gupta et al. (2017), Parisotto & Salakhutdinov (2018) and Henriques & Vedaldi (2018), though we make some simpliﬁcations.”
Learning to Visually Navigate in Photorealistic Environments Without any Supervision (2020, unknown): “Other works replace SLAM entirely by deep models with no planning but explicit map-like or SLAM inspired memory structures [16, 22, 23, 36].”
Memory Augmented Control Networks (2017, background): “Other works that attempt to solve planning in partially observable environments solve this by augmenting the environment with markers to help the agent [16],[22].”
Continual and Multi-task Reinforcement Learning With Shared Episodic Memory (2019, background): “Both of these features of adaptive behavior are being actively studied in the ﬁeld of reinforcement learning (Rusu et al., 2015; Chaplot et al., 2017; Teh et al., 2017; Oh et al., 2016; Parisotto & Salakhutdinov, 2017; Santoro et al., 2018; Schwarz et al., 2018).”
Learning Object-conditioned Exploration using Distributed Soft Actor Critic (2020, background): “Researchers have studied a wide variety of navigation problems, such as metric navigation [9, 10, 11, 12], targetdriven semantic navigation [7, 13, 14], and exploration [15, 12, 16, 17, 18, 19].”
Navigation and planning in latent maps (2018, methodology): “This prior knowledge was then added to the architecture of the agent’s policy by (Parisotto & Salakhutdinov, 2017).”
A Behavioral Approach to Visual Navigation with Graph Localization Networks (2019, background): “Through this representation, we avoid reliance on metric information [33] or specific robot poses [18].”