Toward understanding natural language directions

By Thomas Kollar, Stefanie Tellex, Deb K. Roy, Nicholas A. Roy. 2010

458 citations recorded by the source corpus.

Speaking using unconstrained natural language is an intuitive and flexible way for humans to interact with robots. Understanding this kind of linguistic input is challenging because diverse words and phrases must be mapped into structures that the robot can understand, and elements in those structures must be grounded in an uncertain environment. We present a system that follows natural language directions by extracting a sequence of spatial description clauses from the linguistic input and then infers the most probable path through the environment given only information about the environmental geometry and detected visible objects. We use a probabilistic graphical model that factors into three key components. The first component grounds landmark phrases such as “the computers” in the perceptual frame of the robot by exploiting co-occurrence statistics from a database of tagged images such as Flickr. Second, a spatial reasoning component judges how well spatial relations such as “past the computers” describe a path. Finally, verb phrases such as “turn right” are modeled according to the amount of change in orientation in the path. Our system follows 60% of the directions in our corpus to within 15 meters of the true destination, significantly outperforming other approaches.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Language Understanding for Text-based Games using Deep Reinforcement Learning (2015, background): “, 2010), navigating with directions (Vogel and Jurafsky, 2010; Kollar et al., 2010; Artzi and Zettlemoyer, 2013; Matuszek et al., 2013; Andreas and Klein, 2015) and playing computer games (Eisenstein et al.”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “There is a long history of research studying how to connect language and behavior [46, 47, 48, 49, 50].”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Some works utilize language for planning and grounding [39, 40, 41, 42, 43, 44].”
LILA: Language-Informed Latent Actions (2021, background): “A variety of methods have sought to combine language and robotics, spanning approaches that map language to planning primitives [9, 30, 31, 32, 33], perform imitation learning from demonstrations and instructions [11, 34, 35, 36], and pair language instructions with reward functions for…”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “This is typically achieved either by assuming that the set of all navigation goals, or objects to be acted upon, has been enumerated, and that each will be identified by label [11, 43], or by operating in severely restricted or simulated environments requiring limited perception [10, 16, 19, 24, 29,”
Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions (2013, methodology): “The lexicon contains entries that pair words or phrases with categories.”
Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation (2011, methodology): “The system computes word-label similarity in two ways: using WordNet; and from co-occurrence statistics obtained by downloading millions of images and corresponding tags from Flickr (Kollar et al., 2010).”
Learning to Interpret Natural Language Navigation Instructions from Observations (2011, methodology): “In particular, we focus on the task of navigation (MacMahon, Stankiewicz, and Kuipers 2006; Shimizu and Haas 2009; Matuszek, Fox, and Koscher 2010; Kollar et al. 2010; Vogel and Jurafsky 2010).”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “In a typical instruction following problem, the agent is given a description of the goal state or of a preferred policy as a proxy for a description of the task [MacMahon et al. , 2006; Kollar et al. , 2010].”
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences (2015, background): “[19] build a generative model over the assumed flat, sequential structure of language that includes a combination of pre-specified and learned models for spatial relations, adverbs, and verbs.”
Asking for Help Using Inverse Semantics (2014, background): “Forward semantics is the problem of mapping between words in language and aspects of the external world; the canonical problem is enabling a robot to follow a person’s natural language commands [MacMahon et al., 2006, Kollar et al., 2010, Tellex et al., 2011, Matuszek et al., 2012].”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “However, prior work has often studied 2D environments [24, 32, 27, 52] and simpliﬁed actuators [48, 18, 12].”
Grounding Verbs of Motion in Natural Language Commands to Robots (2010, background): “Previous work focused on natural language route instructions [Kollar et al., 2010, Matuszek et al., 2010, MacMahon et al., 2006, Shimizu and Haas, 2009], using a natural language command to infer a path through an unchanging environment.”
RTFM: Generalising to Novel Environment Dynamics via Reading (2019, background): “Examples of such context include images (Barnard & Forsyth, 2001), games (Chen & Mooney, 2008; Wang et al., 2016), robot control (Kollar et al., 2010; Tellex et al., 2011), and navigation (Anderson et al., 2018).”
Learning Semantic Maps from Natural Language Descriptions (2013, background): “In the robotics domain, the grounding problem has been mainly addressed in the context of following natural language commands [1, 2, 3, 13, 14, 15, 16, 17].”
Representation Learning for Grounded Spatial Reasoning (2017, methodology): “Previous computational approaches include techniques such as proximity fields (Kelleher et al., 2006), spatial templates (Levit and Roy, 2007) and geometrically defined mappings (Moratz and Tenbrink, 2006; Kollar et al., 2010).”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “Many prior works have studied this problem in the context of instruction following, where an agent aims to complete a task specified by formal language/programs [13, 14, 15, 16, 17, 18] or natural language [10, 11, 19, 20].”