Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World

By Jayant Krishnamurthy, Thomas Kollar. 2013

176 citations recorded by the source corpus.

This paper introduces Logical Semantics with Perception (LSP), a model for grounded language acquisition that learns to map natural language statements to their referents in a physical environment. For example, given an image, LSP can map the statement “blue mug on the table” to the set of image segments showing blue mugs on tables. LSP learns physical representations for both categorical (“blue,” “mug”) and relational (“on”) language, and also learns to compose these representations to produce the referents of entire statements. We further introduce a weakly supervised training procedure that estimates LSP’s parameters using annotated referents for entire statements, without annotated referents for individual words or the parse structure of the statement. We perform experiments on two applications: scene understanding and geographical question answering. We find that LSP outperforms existing, less expressive models that cannot represent relational language. We further find that weakly supervised training is competitive with fully supervised training while requiring significantly less annotation effort.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Semantic Parsing on Freebase from Question-Answer Pairs (2013, background): “, 2011), following navigational instructions (Chen, 2012; Artzi and Zettlemoyer, 2013), and interacting in the real world via perception (Matuszek et al., 2012; Tellex et al., 2011; Krishnamurthy and Kollar, 2013).”
A simple neural network module for relational reasoning (2017, background): “gh grounding language in spatial percepts has a long-standing tradition, the majority of previous research has focused on either rule-based spatial representations or hand-engineered spatial features [8, 10, 20, 21, 24, 29, 38, 39]. Although there are some attempts to learn spatial relations using s”
Neural Module Networks (2015, background): “General compositional semantics There is a large literature on learning to answer questions about structured knowledge representations from question–answer pairs, both with and without joint learning of meanings for simple predicates [23, 21].”
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input (2014, background): “Language and perception: Previous work [4, 5] has proposed models for the language grounding problem with the goal of connecting the meaning of the natural language sentences to a perceived world.”
Learning to Compose Neural Networks for Question Answering (2016, methodology): “Our dynamic model (D-NMN) outperforms both the logical (LSP-F) and perceptual models (LSP-W) described by (Krishnamurthy and Kollar, 2013), as well as a fixed-structure neural module net (NMN).”
Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images (2015, background): “While such problems have been historically addressed by symbolic semantic parsing techniques [11, 16], there is a recent trend of machine learning-based approaches [9, 8, 10] to ﬁnd the associations.”
Grounding of Textual Phrases in Images by Reconstruction (2015, background): “Language grounding in visual data is an interesting problem studied both in computer vision [16, 17, 19, 26] and natural language processing communities [20, 25].”
Modeling Biological Processes for Reading Comprehension (2014, background): “In particular, Krishnamurthy and Kollar (2013) learned to map natural language utterances to referents in an image by constructing a KB from the image and then mapping the utterance to a query over the KB.”
Scaling Semantic Parsers with On-the-Fly Ontology Matching (2013, background): “, 2011), relation extraction (Krishnamurthy and Mitchell, 2012), robot control (Matuszek et al., 2012; Krishnamurthy and Kollar, 2013), interpreting instructions (Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013), and generating programs (Kushman and Barzilay, 2013).”
Building a Semantic Parser Overnight (2015, background): “, in regular expressions (Kushman and Barzilay, 2013) or grounding to perception (Matuszek et al., 2012; Tellex et al., 2011; Krishnamurthy and Kollar, 2013).”
Modeling Relationships in Referential Expressions with Compositional Modular Networks (2016, background): “Although [14] also learns language parsing and perception, it is directly based on logic (λ-calculus) and requires additional classifiers trained for each predicate class.”
Towards a Visual Turing Challenge (2014, background): “Most prominently, the latter is about finding (either with a hand-crafted set of rules or learnt-based approaches) a mapping between the linguistic fragments and the physical world [14, 15, 41], whereas the question answering task is about an end-to-end system where we do not necessarily want to enf”
FVQA: Fact-Based Visual Question Answering (2016, methodology): “However, our approach differs from [62] in that it focuses on visual questions requiring support from both physical environment and external commonsense knowledge, which predicts over a much larger, multi-domain KB.”
Learning executable semantic parsers for natural language understanding (2016, background): “[29, 2], identifying objects in a scene [23, 15], converting nat-ural language to regular expressions [17], and many others.”
Large-scale Semantic Parsing without Question-Answer Pairs (2014, methodology): “…feedback-based learning has been previously used for interpreting and following NL instructions (Branavan et al., 2009; Chen and Mooney, 2011), playing computer games (Branavan et al., 2012), and grounding language in the physical world (Krishnamurthy and Kollar, 2013; Matuszek et al., 2012).”
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering (2016, background): “While such problems have been historically addressed by symbolic semantic parsing techniques (Krishnamurthy and Kollar, 2013; Matuszek et al, 2012), there is a recent trend of machine learning-based approaches (Karpathy and Fei-Fei, 2015; Karpathy et al, 2014; Akata et al, 2016; Kong et al, 2014;…”
Probabilistic Models for Learning a Semantic Parser Lexicon (2016, background): “The heuristics often take the form of lexicon templates, which are rules that create lexicon entries by pattern-matching training examples (Liang et al., 2011; Krishnamurthy and Mitchell, 2012; Krishnamurthy and Kollar, 2013).”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, unknown): “Some Visual QA works indeed extract scene graphs followed by applying algorithms designed for reasoning on curated KBs [40, 64].”