Learning models for following natural language directions in unknown environments

By Sachithra Hemachandra, Felix Duvallet, Thomas M. Howard, Nicholas A. Roy, Anthony Stentz, Matthew R. Walter. 2015

93 citations recorded by the source corpus.

Natural language offers an intuitive and flexible means for humans to communicate with the robots that we will increasingly work alongside in our homes and workplaces. Recent advancements have given rise to robots that are able to interpret natural language manipulation and navigation commands, but these methods require a prior map of the robot's environment. In this paper, we propose a novel learning framework that enables robots to successfully follow natural language route directions without any previous knowledge of the environment. The algorithm utilizes spatial and semantic information that the human conveys through the command to learn a distribution over the metric and semantic properties of spatially extended environments. Our method uses this distribution in place of the latent world model and interprets the natural language instruction as a distribution over the intended behavior. A novel belief space planner reasons directly over the map and behavior distributions to solve for a policy using imitation learning. We evaluate our framework on a voice-commandable wheelchair. The results demonstrate that by learning and performing inference over a latent environment model, the algorithm is able to successfully follow natural language route directions within novel, extended environments.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

End-to-End Driving Via Conditional Imitation Learning (2017, background): “Human guidance of robot actions has been studied extensively [5], [14], [24], [34], [35].”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “Embodied Navigation Agent Navigation in 3D environments [50, 25, 26, 14] is an essential capability of a mobile intelligent system that functions in the physical world.”
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences (2015, neutral): “form navigational route instructions [20, 19, 6, 5, 15, 16, 12].”
Robust Navigation with Language Pretraining and Stochastic Sampling (2019, background): “The vision-and-language navigation (VLN) task, learning to navigate in visual environments based on natural language instructions, has attracted interest throughout the artiﬁcial intelligence research community (Hemachandra et al., 2015; Anderson et al., 2018; Chen et al., 2019; Savva et al., 2019).”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “…& Steels, 2015; Dubba et al., 2014), embodied question answering (Das et al., 2018; Gordon et al., 2018), and navigation (Matuszek et al., 2013; Hemachandra et al., 2015; Duvallet et al., 2016; Zhu et al., 2017; de Vries et al., 2018; Yuke Zhu, 2017; Mousavian et al., 2018; Wayne et al., 2018;…”
Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling (2019, background): “Vision-and-Language Navigation Navigation in 3D environments based on natural language instruction has recently been investigated by many studies [3, 8, 21, 22, 26, 20, 33, 25, 11, 32, 31, 16].”
Real-time natural language corrections for assistive robotic manipulators (2017, background): “Duvallet et al. (2016) and Hemachandra et al. (2015) describe approaches for following directions in partially observed environments, and Mei et al. (2016) develop a neural sequence-to-sequence model to translate natural language instructions to action sequences.”
Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learning (2018, background): “Mapping natural language instructions to actions has been studied extensively, both using physical robots [7, 16, 25, 31, 34, 49, 50] and virtual agents [2, 4, 29, 30, 32].”
Transferable Representation Learning in Vision-and-Language Navigation (2019, background): “Embodied Agent Navigation Navigation in realistic 3D environments has also received increased interest recently [35, 18, 29, 46].”
Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight (2019, background): “Natural language instruction following has been extensively studied using hand-engineered symbolic intermediate representations of world state or instruction semantics with physical robots [1, 2, 3, 4, 17, 5, 18, 6, 19] and simulated agents [20, 21, 22, 23, 24, 25].”
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following (2020, background): “Natural language instruction following on physical robots is most commonly studied using handengineered symbolic representations of world state or instruction semantics [1, 7, 2, 3, 4, 8, 5, 9], which require representation design and state estimation pipelines that are hard to scale to complex envi”