Grounding spatial relations for human-robot interaction

By Sergio Guadarrama, Lorenzo Riano, Dave Golland, Daniel Goehring, Yangqing Jia, Daniel Klein, Pieter Abbeel, Trevor Darrell. 2013

147 citations recorded by the source corpus.

We propose a system for human-robot interaction that learns both models for spatial prepositions and for object recognition. Our system grounds the meaning of an input sentence in terms of visual percepts coming from the robot's sensors in order to send an appropriate command to the PR2 or respond to spatial queries. To perform this grounding, the system recognizes the objects in the scene, determines which spatial relations hold between those objects, and semantically parses the input sentence. The proposed system uses the visual and spatial information in conjunction with the semantic parse to interpret statements that refer to objects (nouns), their spatial relationships (prepositions), and to execute commands (actions). The semantic parse is inherently compositional, allowing the robot to understand complex commands that refer to multiple objects and relations such as: “Move the cup close to the robot to the area in front of the plate and behind the tea box”. Our system correctly parses 94% of the 210 online test sentences, correctly interprets 91% of the correctly parsed sentences, and correctly executes 89% of the correctly interpreted sentences.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

A simple neural network module for relational reasoning (2017, background): “gh grounding language in spatial percepts has a long-standing tradition, the majority of previous research has focused on either rule-based spatial representations or hand-engineered spatial features [8, 10, 20, 21, 24, 29, 38, 39]. Although there are some attempts to learn spatial relations using s”
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input (2014, background): “We realize that the skilled use of spatial relations is a complex task and grounding spatial relations is a research thread on its own (e.g. [17], [18] and [19]).”
Gated-Attention Architectures for Task-Oriented Language Grounding (2017, background): “Some of the work that aims to ground navigational instructions include Guadarrama et al. (2013), Bollini et al. (2013) and Beetz et al. (2011), where the focus was to ground verbs like go, follow, etc and spatial relations of verbs (Tellex et al., 2011; Fasola and Mataric, 2013).”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, background): “Many works have tackled this problem [37,38,39,40], enforcing that language terms be grounded in the image.”
Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions (2014, methodology): “Other than the works discussed in the introduction (Beetz et al., 2011; Bollini et al., 2012; Guadarrama et al., 2013; Marco et al., 2012), the problem of navigation has been addressed by using learned models for verbs such as follow, meet, go, and dynamic spatial relations such as walk close to…”
Open-vocabulary Object Retrieval (2014, background): “In particular, we define a set of functions Φ = {φj}, j = 1, ...,m, that project a given image ci into a sparse vector of words, i.e.”
Learning Interpretable Spatial Operations in a Rich 3D Blocks World (2017, background): “2014), spatial reasoning (Steels and Vogt 1997; Roy 2002; Guadarrama et al. 2013), and action taking (MacMahon, Stankiewicz, and Kuipers 2006; Yu and Siskind 2013).”
Optimization Beyond the Convolution: Generalizing Spatial Relations with End-to-End Metric Learning (2017, background): “robot interaction [4], [5], [6] and active object search in complex environments [7].”
Metric learning for generalizing spatial relations to new objects (2017, methodology): “Guadarrama et al. learn models of pre-deﬁned prepositions by training a multi-class logistic regression model using data gathered from crowdsourcing [6].”
A Pooling Approach to Modelling Spatial Relations for Image Retrieval and Annotation (2014, methodology): “[17] uses only 2 spatial prepositions, while [9] and [10] concentrate on 11.”
Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms (2018, unknown): “Guadarrama et al. (2013) demonstrated a system that integrates visual and spatial information with semantic parsing to interpret actions and ground spatial relations.”
Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction (2018, background): “Again, some earlier work treats relational grounding as a classiﬁcation task with predeﬁned relation templates [9, 10, 14].”
Environment-Driven Lexicon Induction for High-Level Instructions (2015, unknown): “The task of mapping natural language instructions to actions for a robot has been gaining momentum in recent years (Artzi and Zettlemoyer, 2013; Tellex et al., 2011; Misra et al., 2014; Bollini et al., 2011; Guadarrama et al., 2013; Matuszek et al., 2012b; Fasola and Mataric, 2013).”
Towards a Dataset for Human Computer Communication via Grounded Language Acquisition (2016, background): “2014) and on learning spatial relations and actions for small vocabularies with hard-coded abstract concepts (Steels and Vogt 1997; Roy 2002; Guadarrama et al. 2013).”
Natural Language Communication with Robots (2016, background): “Progress on language in this area has largely focused on grounding visual attributes (Kollar et al., 2013; Matuszek et al., 2014) and on learning spatial relations and actions for small vocabularies with hard-coded abstract concepts (Steels and Vogt, 1997; Roy, 2002; Guadarrama et al., 2013).”