A framework for learning semantic maps from grounded natural language descriptions

By Matthew R. Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, Seth J. Teller. 2014

51 citations recorded by the source corpus.

This paper describes a framework that enables robots to efficiently learn human-centric models of their environment from natural language descriptions. Typical semantic mapping approaches are limited to augmenting metric maps with higher-level properties of the robot’s surroundings (e.g. place type, object locations) that can be inferred from the robot’s sensor data, but do not use this information to improve the metric map. The novelty of our algorithm lies in fusing high-level knowledge that people can uniquely provide through speech with metric information from the robot’s low-level sensor streams. Our method jointly estimates a hybrid metric, topological, and semantic representation of the environment. This semantic graph provides a common framework in which we integrate information that the user communicates (e.g. labels and spatial relations) with metric observations from low-level sensors. Our algorithm efficiently maintains a factored distribution over semantic graphs based upon the stream of natural language and low-level sensor information. We detail the means by which the framework incorporates knowledge conveyed by the user’s descriptions, including the ability to reason over expressions that reference yet unknown regions in the environment. We evaluate the algorithm’s ability to learn human-centric maps of several different environments and analyze the knowledge inferred from language and the utility of the learned maps. The results demonstrate that the incorporation of information from free-form descriptions increases the metric, topological, and semantic accuracy of the recovered environment model.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Grounded Language Learning in a Simulated 3D World (2017, background): “Within robotics, there has been interest in using language to facilitate human-robot communication, as part of which it is necessary to devise mechanisms for grounding a perceptible environment with language (Hemachandra et al., 2014; Walter et al., 2014).”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, methodology): “Both object identiﬁcation and manipulation are likely far harder in reality, and it remains to be seen whether our proposed method would work seamlessly for robot language understanding [Tellex et al., 2011, 2012, Walter et al., 2014].”
Factor Graphs for Robot Perception (2017, background): “7 Summary It is clear that the above is not meant to be a comprehensive review, and factor graphs have been used in a variety of other settings, from calibration [126] and occupancy grid mapping [51] to connecting language to robotic perception and action [195].”
Multimodal Visual and Simulated Muscle Activations for Grounded Semantics of Hand-related Descriptions (2018, background): “…Roy, 2005) and recent work in grounded semantic learning in various tasks and settings, notably learning descriptions of the immediate environment (Walter et al., 2014); navigation (Kollar et al., 2010); nouns, adjectives, and relational spatial descriptions (Kennington and Schlangen, 2015);…”