Natural Language Acquisition and Grounding for Embodied Robotic Systems

By Muhannad Al-Omari, Paul Duckworth, David C. Hogg, Anthony G. Cohn. 2017

54 citations recorded by the source corpus.

We present a cognitively plausible novel framework capable of learning the grounding in visual semantics and the grammar of natural language commands given to a robot in a table top environment. The input to the system consists of video clips of a manually controlled robot arm, paired with natural language commands describing the action. No prior knowledge is assumed about the meaning of words, or the structure of the language, except that there are different classes of words (corresponding to observable actions, spatial relations, and objects and their observable properties). The learning process automatically clusters the continuous perceptual spaces into concepts corresponding to linguistic input. A novel relational graph representation is used to build connections between language and vision. As well as the grounding of language to perception, the system also induces a set of probabilistic grammar rules. The knowledge learned is used to parse new commands involving previously unseen objects.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Towards Automatic Learning of Procedures From Web Instructional Videos (2017, background): “2016)—learning the steps of accomplishing these tasks and their dependencies is essential, especially for agents’ automatic acquisition of language or manipulation skills from video (Yu and Siskind 2013; Al-Omari et al. 2017; Yu and Siskind 2015).”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “…for a particular goal such as object manipulation with instructions (Misra et al., 2016; Arkin et al., 2017), grounded language acquisition (Al-Omari et al., 2017; Kollar et al., 2013; Spranger & Steels, 2015; Dubba et al., 2014), embodied question answering (Das et al., 2018; Gordon et…”
Grounding Language to Autonomously-Acquired Skills via Goal Generation (2021, unknown): “Several works investigate the use of semantic representations to associate meanings and skills (Alomari et al., 2017; Tellex et al., 2011; Kulick et al., 2013).”
Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions (2018, background): “Kiela and Clark 2015; Thomason et al. 2016; Gao et al. 2016b; Alomari et al. 2017).”
Multimodal Visual and Simulated Muscle Activations for Grounded Semantics of Hand-related Descriptions (2018, background): “Other recent work has already gone beyond visual grounded semantics including olfactory perception (Kiela et al., 2015), auditory perception (Kiela and Clark, 2015), haptics (Alomari et al., 2017), and multimodal features including haptic, auditory, and proprioceptive (Thomason et al., 2016).”
Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction (2018, background): “This task is often formulated as a visual-semantic alignment problem [7] and has broad applications including retrieval [7, 8], description generation [20, 25], and human-robot interaction [1, 22].”
Robots That Use Language (2020, background): “Some work has focused specifically on spatial prepositions (11, 127, 169, 170).”
Opportunistic Active Learning for Grounding Natural Language Descriptions (2017, background): “Finally, by performing more complicated language understanding than simple stopword removal, future work may also be able to learn to jointly understand language predicates and human commands (as in [21, 22, 23]), additionally leveraging opportunistic active learning to ask questions that best clari”
Grounding language acquisition by training semantic parsers using captioned videos (2018, background): “Al-Omari et al. (2017) acquire a grammar for a fragment of English and Arabic from videos paired with sentences.”