Learning Multi-Modal Grounded Linguistic Semantics by Playing "I Spy"

By Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, Raymond J. Mooney. 2016

104 citations recorded by the source corpus.

Subfields: Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Experience Grounds Language (2020, background): “Further, tactile senses lend meaning, both physical (Sinapov et al., 2014; Thomason et al., 2016) and abstract, to concepts like heavy and soft.”
PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “Physical understanding is broad domain that touches on everything from scientiﬁc knowledge (Schoenick et al. 2016) to the interactive acquisition of knowledge by embodied agents (Thomason et al. 2016).”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “By operating only in simulation, our model also misses the full range of experience that can ground language in the world [11], such as haptic feedback during object manipulation [78, 79, 68], and audio [16] and speech [31, 41] features of the environment.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Robots can use a wide range of sensor modalities to observe the effects of its interactions, including haptic, tactile, vision, and audio [235, 63, 159, 128, 388].”
Language to Action: Towards Interactive Task Learning with Physical Agents (2018, background): “…to perception involves connecting meanings of words to machine perception [Roy, 2005; Matuszek et al. , 2012; Kennington and Schlangen, 2015; Thomason et al. , 2016] and grounding language expressions to visual objects [Liu et al. , 2012; Williams and Scheutz, 2018], to physical landmarks…”
Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions (2018, background): “Past work has demonstrated that using sensory data beyond vision to ground language predicates improves robotic performance over using vision alone (Thomason et al. 2016).”
Collaborative Language Grounding Toward Situated Human-Robot Dialogue (2017, result): “As shown in this article and other recent work (Cantrell et al. 2012; Mohan, Kirk, and Laird 2013; Mohseni-Kabir et al. 2015; Thomason et al. 2016), language and collaborative dialogue play an impor- 42 AI MAGAZINE WINTER 2016 43 tant role in enabling the robot to continuously learn grounded…”
Opportunistic Active Learning for Grounding Natural Language Descriptions (2017, background): “Past work has shown that using non-visual modalities when performing language grounding can help with non-visual words like “heavy,” and we follow that work’s methodology for training and ensembling SVM classifiers for each predicate to predict whether that predicate applies to a novel object [11].”
Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog (2020, methodology): “Previous work on learning to ground object attributes and names using dialog framed the data gathering phase as a 20 Questions -style [40] or I Spy [28, 38] game.”
Learning Interpretable Spatial Operations in a Rich 3D Blocks World (2017, unknown): “For example, recent work (Thomason et al. 2015; 2016; 2017) has shown how these instructions can be combined with exploration on physical robotics to follow instructions and learn representations online.”
Multimodal Visual and Simulated Muscle Activations for Grounded Semantics of Hand-related Descriptions (2018, background): “Other recent work has already gone beyond visual grounded semantics including olfactory perception (Kiela et al., 2015), auditory perception (Kiela and Clark, 2015), haptics (Alomari et al., 2017), and multimodal features including haptic, auditory, and proprioceptive (Thomason et al., 2016).”
Language Grounding with 3D Objects (2021, background): “The physical forces and sounds objects make during manipulation actions can also be associated with words such as rattling and heavy for multimodal understanding beyond vision [19, 37].”