MapNet: An Allocentric Spatial Memory for Mapping Environments

By João F. Henriques, Andrea Vedaldi. 2018

181 citations recorded by the source corpus.

Autonomous agents need to reason about the world beyond their instantaneous sensory input. Integrating information over time, however, requires switching from an egocentric representation of a scene to an allocentric one, expressed in the world reference frame. It must also be possible to update the representation dynamically, which requires localizing and registering the sensor with respect to the world reference. In this paper, we develop a differentiable module that satisfies such requirements, while being robust, efficient, and suitable for integration in end-to-end deep networks. The module contains an allocentric spatial memory that can be accessed associatively by feeding to it the current sensory input, resulting in localization, and then updated using an LSTM or similar mechanism. We formulate efficient localization and registration of sensory information as a dual pair of convolution/deconvolution operators in memory space. The map itself is a 2.5D representation of an environment storing information that a deep neural network module learns to distill from RGBD input. The result is a map that contains multi-task information, different from classical approaches to mapping such as structure-from-motion. We present results using synthetic mazes, a dataset of hours of recorded gameplay of the classic game Doom, and the very recent Active Vision Dataset of real images captured from a robot.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “In this work, the authors build upon MapNet [55] to include 2.”
Cognitive Mapping and Planning for Visual Navigation (2017, background): “to the real world [12,64], and how the domain gap between simulation and the real world may be reduced [76]. A number of works have studied related navigation problems in such simulation environments [6,15,36]. Researchers have also gone beyond specifying goals as a desired location in space and ﬁnd”
Learning Exploration Policies for Navigation (2019, methodology): “Some of these design choices were inspired by recent work from Gupta et al. (2017), Parisotto & Salakhutdinov (2018) and Henriques & Vedaldi (2018), though we make some simpliﬁcations.”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “Methods also investigate the utility of topological and latent memory maps [35,78,37,99], graph-”
Benchmarking Classic and Learned Navigation in Complex 3D Environments (2019, background): “In contrast, new learning-based methods are usually evaluated in specialized simulated environments [31, 37, 16, 18, 44].”
Simultaneous Mapping and Target Driven Navigation (2019, methodology): “[10] proposed an architecture that dynamically updates an agent’s allocentric representation for the task of localization.”
Audio-Visual Embodied Navigation (2019, background): “Following these ﬁndings, recent visual AI agents aggregate egocentric visual inputs [118, 117, 65, 90], often with a spatio-temporal memory [41, 81, 46, 105, 98].”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “Recent AI agents also aggregate egocentric visual inputs [112,111,63,86,46], often with a spatio-temporal memory [40,79,44,101].”
Object Goal Navigation using Goal-Oriented Semantic Exploration (2020, methodology): “These include methods which use recurrent neural networks [30, 25, 6, 37, 21, 7, 38, 42], structured spatial representations [16, 33, 8, 19, 15] and topological representations [35, 36].”
Predicting Semantic Map Representations From Images Using Pyramid Occupancy Networks (2020, background): “[13] take advantage of RGB-D images to learn an implicit map representation which can be used for later localisation.”
Learning to Visually Navigate in Photorealistic Environments Without any Supervision (2020, unknown): “This map has been further extended with image features [14] or with a dynamic structure [2, 15].”
DeepVoxels: Learning Persistent 3D Feature Embeddings (2018, methodology): “MapNet [18] performs SLAM based on a scene-specific 2D feature grid representation.”
Learning to Explore using Active Neural SLAM (2020, background): “Researchers have designed neural network policies that reason via spatial representations (Gupta et al., 2017; Parisotto and Salakhutdinov, 2018; Zhang et al., 2017; Henriques and Vedaldi, 2018; Gordon et al., 2018), topological representations (Savinov et al.”
An Exploration of Embodied Visual Exploration (2020, unknown): “regates information to facilitate long-term information storage and effective planning. Unlike traditional SLAM, such a learned spatio-temporal memory—popular in recent embodied perception approaches [21, 26, 14, 13]—allows the agent to leverage both statistical visual patterns as well as geometry t”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “Instead, recent work in embodied visual navigation explores learned maps that leverage both visual patterns as well as geometry, with the advantage of extrapolating to novel environments ( e.g ., [23, 22, 60, 26, 10]).”
Chasing Ghosts: Instruction Following as Bayesian State Tracking (2019, methodology): “Semantic spatial maps are grid-based representations containing convolutional neural network (CNN) features which have been recently proposed in the context of visual navigation [9], interactive question answering [12], and localization [11].”
Occupancy Anticipation for Efficient Exploration and Navigation (2020, background): “One of the key factors for success in navigation has been the movement towards complex map-based architectures [20,46,11,10] that capture both geometry [20,11,10] and semantics [20,46,19,24], thereby facilitating efficient policy learning and planning.”