Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation

By Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, A. Banerjee, Seth J. Teller, Nicholas A. Roy. 2011

874 citations recorded by the source corpus.

This paper describes a new model for understanding natural language commands given to autonomous systems that perform navigation and mobile manipulation in semi-structured environments. Previous approaches have used models with fixed structure to infer the likelihood of a sequence of actions given the environment and the command. In contrast, our framework, called Generalized Grounding Graphs, dynamically instantiates a probabilistic graphical model for a particular natural language command according to the command's hierarchical and compositional semantic structure. Our system performs inference in the model to successfully find and execute plans corresponding to natural language commands such as "Put the tire pallet on the truck." The model is trained using a corpus of commands collected using crowdsourcing. We pair each command with robot actions and use the corpus to learn the parameters of the model. We evaluate the robot's performance by inferring plans from natural language commands, executing each plan in a realistic robot simulator, and asking users to evaluate the system's performance. We demonstrate that our system can successfully follow many natural language commands from the corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Experience Grounds Language (2020, background): “world (Tellex et al., 2011; Matuszek, 2018; Thomason et al., 2020; Tellex et al., 2020) must translate from language to action.”
PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “Language tools should provide strong initial priors for learning (Tellex et al. 2011; Matuszek 2018) that are then reﬁned through interaction and dialogue (Gao et al. 2016).”
Semantic Parsing on Freebase from Question-Answer Pairs (2013, background): “…Lei et al., 2013), playing computer games (Branavan et al., 2010; Branavan et al., 2011), following navigational instructions (Chen, 2012; Artzi and Zettlemoyer, 2013), and interacting in the real world via perception (Matuszek et al., 2012; Tellex et al., 2011; Krishnamurthy and Kollar, 2013).”
A simple neural network module for relational reasoning (2017, background): “gh grounding language in spatial percepts has a long-standing tradition, the majority of previous research has focused on either rule-based spatial representations or hand-engineered spatial features [8, 10, 20, 21, 24, 29, 38, 39]. Although there are some attempts to learn spatial relations using s”
Learning to Understand Goal Specifications by Modelling Reward (2018, background): “Tellex et al. (2011) train a log-linear model to map instruction constituents into their groundings, which can be objects, places, state sequences, etc.”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “There is a long history of research studying how to connect language and behavior [46, 47, 48, 49, 50].”
Grammar-Based Grounded Lexicon Learning (2022, background): “There have also been approaches for learning grammatical structures from grounded texts [35, 50, 22, 7, 33, 43].”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Some works utilize language for planning and grounding [39, 40, 41, 42, 43, 44].”
LILA: Language-Informed Latent Actions (2021, methodology): “This is enabled by our [ Right ] language-informed latent actions (LILA) models that use auto-encoders to learn language & state-conditioned low-DoF latent spaces for meaningful control. robotics settings for similar purposes within the full autonomy paradigm [9, 10, 11, 12, 13].”
Imitating Interactive Intelligence (2020, background): “In some sense, robotics is the ultimate integrated, interactive research platform (see e.g. Tellex et al. (2011) for a pioneering study of language understanding in robotics).”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “This is typically achieved either by assuming that the set of all navigation goals, or objects to be acted upon, has been enumerated, and that each will be identified by label [11, 43], or by operating in severely restricted or simulated environments requiring limited perception [10, 16, 19, 24, 29,”
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input (2014, background): “Integrated systems that execute commands : Others [9, 10, 11, 12, 13] focus on the task of learning the representation of natural language in the restricted setting of executing commands.”
Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions (2013, methodology): “Recently, Tellex et al. (2011) used a graphical model semantics representation to learn from instructions paired with demonstrations.”
A Joint Model of Language and Perception for Grounded Attribute Learning (2012, background): “Most closely related to our work are approaches that learn probabilistic language models from natural language input (Matuszek et al., 2012; Chen & Mooney, 2011), especially those that include a visual component (Tellex et al., 2011).”
Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World (2013, methodology): “This problem has been addressed in the context of robot direction following (Kollar et al., 2010; Tellex et al., 2011) and visual attribute learning (Matuszek et al.”
End-to-End Driving Via Conditional Imitation Learning (2017, background): “Human guidance of robot actions has been studied extensively [5], [14], [24], [34], [35].”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Systems that learn to carry out natural language instructions in an interactive environment include approaches based on intermediate structured and executable representations of language [50, 9, 4, 28, 20] and approaches that map directly from language and world state observations to actions [7, 2, ”
Learning Language Games through Interaction (2016, background): “Examples include playing games (Branavan et al., 2009, 2010; Reckman et al., 2010) interacting with robotics (Tellex et al., 2011, 2014), and following instructions (Vogel and Jurafsky, 2010; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013) Semantic parsing utterances to logical forms, which we…”