Learning Semantic Maps from Natural Language Descriptions

By Matthew R. Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, Seth J. Teller. 2013

111 citations recorded by the source corpus.

This paper proposes an algorithm that enables robots to efficiently learn human-centric models of their environment from natural language descriptions. Typical semantic mapping approaches augment metric maps with higher-level properties of the robot’s surroundings (e.g., place type, object locations), but do not use this information to improve the metric map. The novelty of our algorithm lies in fusing high-level knowledge, conveyed by speech, with metric information from the robot’s low-level sensor streams. Our method jointly estimates a hybrid metric, topological, and semantic representation of the environment. This semantic graph provides a common framework in which we integrate concepts from natural language descriptions (e.g., labels and spatial relations) with metric observations from low-level sensors. Our algorithm efficiently maintains a factored distribution over semantic graphs based upon the stream of natural language and low-level sensor information. We evaluate the algorithm’s performance and demonstrate that the incorporation of information from natural language increases the metric, topological and semantic accuracy of the recovered environment model.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Natural Language Processing, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Robotics & Control

Citations in the corpus

End-to-End Driving Via Conditional Imitation Learning (2017, background): “Human guidance of robot actions has been studied extensively [5], [14], [24], [34], [35].”
Learning models for following natural language directions in unknown environments (2015, methodology): “directions [2], [3], [4], [5], that command manipulation [6], [7], and that convey environment knowledge [8], [9].”
Learning to Explore using Active Neural SLAM (2020, background): “Research also focuses on incorporating semantics in SLAM (Pronobis and Jensfelt, 2012; Walter et al., 2013).”
Representation Learning for Grounded Spatial Reasoning (2017, methodology): “More recent work in robotics has integrated text containing position information with spatial models of the environment to obtain accurate maps for navigation (Walter et al., 2013; Hemachandra et al., 2014).”
Grounding Robot Plans from Natural Language Instructions with Incomplete World Knowledge (2018, background): “Contemporary language grounding models [1, 3, 4, 18] estimate correspondences between linguistic constituents and the semantic entities observed in the world model but do not directly incorporate background relational knowledge in inference.”
A framework for learning semantic maps from grounded natural language descriptions (2014, methodology): “We describe an approach first presented by the authors (Walter et al., 2013) that enables robots to efficiently learn human-centric models of the environment from a narrated, guided tour (Figure 1) by fusing knowledge inferred from natural language descriptions with conventional low-level sensor dat”
Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions (2014, background): “, 2012), constructing semantic maps (Walter et al., 2013), and high degree of freedom arm planners (e.”
Autonomy in Rehabilitation Robotics: An Intersection (2018, unknown): “For example, there are interfaces that employ haptic feedback to guide the user’s trajectory or alert them to obstacles (74, 75), touch-based graphical user interfaces to capture control commands (76) or locations of interest (65, 77), control command selections via blinking diodes and EEG (78), and”
Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation (2020, background): “Some works formulate active SLAM as Partially Observable Markov Decision Process and use either Bayesian Optimization [27] or Reinforcement Learning [23] to plan trajectories that lead to accurate maps.”
Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight (2019, background): “Natural language instruction following has been extensively studied using hand-engineered symbolic intermediate representations of world state or instruction semantics with physical robots [1, 2, 3, 4, 17, 5, 18, 6, 19] and simulated agents [20, 21, 22, 23, 24, 25].”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, methodology): “When applying these spatial concepts to reasoning, prior works often focus on explicit spatial scene memories [26, 27] or spatial scene graphs [29, 30, 59, 60].”
Deep Transfer in Reinforcement Learning by Language Grounding (2017, background): “Another related line of work consists of systems that learn spatial and topographical maps of the environment for robot navigation using natural language descriptions (Walter et al., 2013; Hemachandra et al., 2014).”
Learning spatial-semantic representations from natural language descriptions and scene classifications (2014, methodology): “[11], our updated representation maintains a distribution over a larger set of semantic properties associated with the environment.”
Visual Place Recognition: A Survey (2016, background): “The addition of semantic data to maps can improve planning and navigation tasks [65] and requires place recognition to be”
Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learning (2018, background): “Mapping natural language instructions to actions has been studied extensively, both using physical robots [7, 16, 25, 31, 34, 49, 50] and virtual agents [2, 4, 29, 30, 32].”