Grounding Verbs of Motion in Natural Language Commands to Robots

By Thomas Kollar, Stefanie Tellex, Deb K. Roy, Nicholas A. Roy. 2010

82 citations recorded by the source corpus.

To be useful teammates to human partners, robots must be able to follow spoken instructions given in natural language. An important class of instructions involve interacting with people, such as “Follow the person to the kitchen” or “Meet the person at the elevators.” These instructions require that the robot fluidly react to changes in the environment, not simply follow a pre-computed plan. We present an algorithm for understanding natural language commands with three components. First, we create a cost function that scores the language according to how well it matches a candidate plan in the environment, defined as the log-likelihood of the plan given the command. Components of the cost function include novel models for the meanings of motion verbs such as “follow,” “meet,” and “avoid,” as well as spatial relations such as “to” and landmark phrases such as “the kitchen.” Second, an inference method uses this cost function to perform forward search, finding a plan that matches the natural language command. Third, a high-level controller repeatedly calls the inference method at each timestep to compute a new plan in response to changes in the environment such as the movement of the human partner or other people in the scene. When a command consists of more than a single task, the controller switches to the next task when an earlier one is satisfied. We evaluate our approach on a set of example tasks that require the ability to follow both simple and complex natural language commands.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Some works utilize language for planning and grounding [39, 40, 41, 42, 43, 44].”
Learning Language Games through Interaction (2016, background): “…2014), and following instructions (Vogel and Jurafsky, 2010; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013) Semantic parsing utterances to logical forms, which we leverage, plays an important role in these settings (Kollar et al., 2010; Matuszek et al., 2012; Artzi and Zettlemoyer, 2013).”
Grounding Predicates through Actions (2021, background): “Other interesting avenues of work include using natural language to learn the pre- and post-conditions of actions [38, 39], or combining visual groundings of symbols with natural language [3, 40].”
Environment-Driven Lexicon Induction for High-Level Instructions (2015, unknown): “Other works (Chen and Mooney, 2011; Kim and Mooney, 2012; Kollar et al., 2010; Fasola and Mataric, 2013) have focused only on navigational verbs and spatial relations, but do not handle highlevel verbs.”
Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions (2014, methodology): “To detail, Kollar et al. (2010) use a maximumlikelihood approach to infer the path taken by the robot.”
Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms (2018, background): “[48] learned cost functions that score motion verbs and spatial relations with candidate plans in the environment.”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “Prior works have looked at using classical tools such as lexical analysis, formal logic, and graphical models to interpret language instructions [27, 7, 6, 26].”