A Joint Model of Language and Perception for Grounded Attribute Learning

By Cynthia Matuszek, Nicholas FitzGerald, Luke S. Zettlemoyer, Liefeng Bo, Dieter Fox. 2012

323 citations recorded by the source corpus.

As robots become more ubiquitous and capable, it becomes ever more important for untrained users to easily interact with them. Recently, this has led to study of the language grounding problem, where the goal is to extract representations of the meanings of natural language tied to the physical world. We present an approach for joint learning of language and perception models for grounded attribute induction. The perception model includes classifiers for physical characteristics and a language model based on a probabilistic categorial grammar that enables the construction of compositional meaning representations. We evaluate on the task of interpreting sentences that describe sets of objects in a physical workspace, and demonstrate accurate task performance and effective latent-variable concept induction in physical grounded scenes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Semantic Parsing on Freebase from Question-Answer Pairs (2013, background): “…Lei et al., 2013), playing computer games (Branavan et al., 2010; Branavan et al., 2011), following navigational instructions (Chen, 2012; Artzi and Zettlemoyer, 2013), and interacting in the real world via perception (Matuszek et al., 2012; Tellex et al., 2011; Krishnamurthy and Kollar, 2013).”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “Finally, joint models of language and perception have also been explored in robotics settings [15].”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “Machine learning has recently been applied to enable abstract task specification, allow general primitives, or relax constraints [67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78].”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “…(Zitnick and Parikh, 2013; Antol et al., 2015), caption generation (Chen et al., 2016), referring expression resolution (Mitchell et al., 2010; Matuszek et al., 2012; FitzGerald et al., 2013), and instruction following (MacMahon et al., 2006; Chen and Mooney, 2011; Bisk et al., 2016; Misra et…”
Neural Module Networks (2015, background): “Outside of question answering, several models have been proposed for instruction following that impose a discrete “planning structure” over an underlying continuous control signal [1, 30].”
Learning to Automatically Solve Algebra Word Problems (2014, background): “…Berant et al., 2013; Kwiatkowski et al., 2013), dialog systems (Artzi and Zettlemoyer, 2011), robot instruction (Chen and Mooney, 2011; Chen, 2012; Kim and Mooney, 2012; Matuszek et al., 2012; Artzi and Zettlemoyer, 2013), and program executions (Kushman and Barzilay, 2013; Lei et al., 2013).”
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input (2014, background): “Language and perception: Previous work [4, 5] has proposed models for the language grounding problem with the goal of connecting the meaning of the natural language sentences to a perceived world.”
Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions (2013, background): “, 2010) and language paired with visual perception (Matuszek et al., 2012).”
Learning to Compose Neural Networks for Question Answering (2016, methodology): “As our approach includes both categories, relations and simple quantification, the range of phenomena considered is generally broader than previous perceptually-grounded QA work (Krishnamurthy and Kollar, 2013; Matuszek et al., 2012).”
Grounding of Textual Phrases in Images by Reconstruction (2015, background): “Language grounding in visual data is an interesting problem studied both in computer vision [16, 17, 19, 26] and natural language processing communities [20, 25].”
Scaling Semantic Parsers with On-the-Fly Ontology Matching (2013, background): “…question answering (QA) (Kwiatkowski et al., 2011; Liang et al., 2011), relation extraction (Krishnamurthy and Mitchell, 2012), robot control (Matuszek et al., 2012; Krishnamurthy and Kollar, 2013), interpreting instruc- tions (Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013), and…”
Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World (2013, methodology): “LSP correctly answers 34% more questions than the most comparable state-of-the-art model (Matuszek et al., 2012).”
Connecting Language and Knowledge Bases with Embedding Models for Relation Extraction (2013, background): “Previous work Learning under weak supervision is common in Natural language processing, especially for tasks where the annotations costs are important such as semantic parsing (Kate and Mooney, 2007; Liang et al., 2009; Bordes et al., 2010; Matuszek et al., 2012).”
Building a Semantic Parser Overnight (2015, background): “, in regular expressions (Kushman and Barzilay, 2013) or grounding to perception (Matuszek et al., 2012; Tellex et al., 2011; Krishnamurthy and Kollar, 2013).”
Learning Language Games through Interaction (2016, background): “…2014), and following instructions (Vogel and Jurafsky, 2010; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013) Semantic parsing utterances to logical forms, which we leverage, plays an important role in these settings (Kollar et al., 2010; Matuszek et al., 2012; Artzi and Zettlemoyer, 2013).”
Towards a Visual Turing Challenge (2014, background): “Mos prominently, the latter is about finding (either with a hand-crafted set of rules or learnt-based approaches) a mapping between the linguistic fragments and the physical world [14, 15, 49], where as the question answering task is about an end-to-end system where we do not necessarily want to enf”
Learning executable semantic parsers for natural language understanding (2016, background): “[29, 2], identifying objects in a scene [23, 15], converting nat-ural language to regular expressions [17], and many others.”
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, methodology): “R2R uses panorama graphs of house environments for the task of navigation instruction following.”