Toward Interactive Grounded Language Acqusition

By Thomas Kollar, Jayant Krishnamurthy, Grant P. Strimel. 2013

41 citations recorded by the source corpus.

—This paper addresses the problem of enabling robots to interactively learn visual and spatial models from multi-modal interactions involving speech, gesture and images. Our approach, called Logical Semantics with Perception (LSP), provides a natural and intuitive interface by signiﬁcantly reducing the amount of supervision that a human is required to provide. This paper demonstrates LSP in an interactive setting. Given speech and gesture input, LSP is able to learn object and relation classiﬁers for objects like mugs and relations like left and right. We extend LSP to generate complex natural language descriptions of selected objects using adjectives, nouns and relations, such as “the orange mug to the right of the green book.” Furthermore, we extend LSP to incorporate determiners (e.g., “the”) into its training procedure, enabling the model to generate acceptable relational language 20% more often than the unaugmented model.

Subfields: Artificial Intelligence, Natural Language Processing, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Robotics & Control

Citations in the corpus

Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “…goal such as object manipulation with instructions (Misra et al., 2016; Arkin et al., 2017), grounded language acquisition (Al-Omari et al., 2017; Kollar et al., 2013; Spranger & Steels, 2015; Dubba et al., 2014), embodied question answering (Das et al., 2018; Gordon et al., 2018), and…”
Grounding language acquisition by training semantic parsers using captioned videos (2018, background): “Kollar et al. (2013) learn to parse similar utterances in an interactive setting.”
Learning Multi-Modal Grounded Linguistic Semantics by Playing "I Spy" (2016, methodology): “Past work has used human speech plus gestures describing sets of objects on a table as supervision to learn attribute classifiers [Matuszek et al., 2014; Kollar et al., 2013].”
A Survey of Current Datasets for Vision and Language Research (2015, background): “…beginning with SHRDLU as one of the first vision-language integration systems (Winograd, 1972) and continuing with more recent attempts on conversational robots grounded in the visual world (Kollar et al., 2013; Cantrell et al., 2010; Matuszek et al., 2012; Kruijff et al., 2007; Roy et al., 2003).”
Visual question answering: A survey of methods and datasets (2016, background): “More recent attempts at creating conversational robotic agents [39, 9, 55, 64] are also grounded in the visual world.”
Learning Interpretable Spatial Operations in a Rich 3D Blocks World (2017, background): “The intersection of robotics and language have seen impressive results in grounding visual attributes (Kollar, Krishnamurthy, and Strimel 2013; Matuszek et al. 2014), spatial reasoning (Steels and Vogt 1997; Roy 2002; Guadarrama et al. 2013), and action taking (MacMahon, Stankiewicz, and Kuipers…”
Opportunistic Active Learning for Grounding Natural Language Descriptions (2017, background): “Turn-taking interactions where humans have to teach the robot concepts using positive and negative labeled examples are typical for interactive language grounding, but do not employ active learning [6, 7, 8, 9].”
Towards a Dataset for Human Computer Communication via Grounded Language Acquisition (2016, background): “Progress on language has largely focused on grounding visual attributes (Kollar, Krishnamurthy, and Strimel 2013; Matuszek et al. 2014) and on learning spatial relations and actions for small vocabularies with hard-coded abstract concepts (Steels and Vogt 1997; Roy 2002; Guadarrama et al. 2013).”
Natural Language Communication with Robots (2016, background): “Progress on language in this area has largely focused on grounding visual attributes (Kollar et al., 2013; Matuszek et al., 2014) and on learning spatial relations and actions for small vocabularies with hard-coded abstract concepts (Steels and Vogt, 1997; Roy, 2002; Guadarrama et al., 2013).”