Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learning

By Valts Blukis, Nataly Brukhim, Andrew Bennett, Ross A. Knepper, Yoav Artzi. 2018

66 citations recorded by the source corpus.

We introduce a method for following high-level navigation instructions by mapping directly from images, instructions and pose estimates to continuous low-level velocity commands for real-time control. The Grounded Semantic Mapping Network (GSMN) is a fully-differentiable neural network architecture that builds an explicit semantic map in the world reference frame by incorporating a pinhole camera projection model within the network. The information stored in the map is learned from experience, while the local-to-world transformation is computed explicitly. We train the model using DAggerFM, a modified variant of DAgger that trades tabular convergence guarantees for improved training speed and memory use. We test GSMN in virtual environments on a realistic quadcopter simulator and show that incorporating an explicit mapping and grounding modules allows GSMN to outperform strong neural baselines and almost reach an expert policy performance. Finally, we analyze the learned map representations and show that using an explicit map leads to an interpretable instruction-following model.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

LILA: Language-Informed Latent Actions (2021, methodology): “Other work in this paradigm that uses imitation learning, but with real-world deployments include works for quadcopter ﬂight [34] and some limited manipulation [13].”
Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight (2019, methodology): “For simulation, we use the quadcopter simulator from Blukis et al. [34] that is based on Microsoft AirSim [43].”
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following (2020, methodology): “Policy Model We use the two-stage policy decomposition of the Position Visitation Network v2 [PVN2; 26, 6]: (a) predict the probability of visiting each position during instruction execution and (b) generate actions that visit high probability positions.”
A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution (2021, background): “Modelling improvements have enabled these approaches to achieve good performance on complex navigation tasks [7, 9, 11, 13, 14, 37], a success that has not yet translated to mobile manipulation [12, 46, 47].”