Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions

By Jesse Thomason, Jivko Sinapov, Raymond J. Mooney, Peter Stone. 2018

35 citations recorded by the source corpus.

A major goal of grounded language learning research is to enable robots to connect language predicates to a robot's physical interactive perception of the world. Coupling object exploratory behaviors such as grasping, lifting, and looking with multiple sensory modalities (e.g., audio, haptics, and vision) enables a robot to ground non-visual words like ``heavy'' as well as visual words like ``red''. A major limitation of existing approaches to multi-modal language grounding is that a robot has to exhaustively explore training objects with a variety of actions when learning a new such language predicate. This paper proposes a method for guiding a robot's behavioral exploration policy when learning a novel predicate based on known grounded predicates and the novel predicate's linguistic relationship to them. We demonstrate our approach on two datasets in which a robot explored large sets of objects and was tasked with learning to recognize whether novel words applied to those objects.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Recent works have studied how to ground modern language models, by training them to accept additional environment inputs [23, 24, 25, 26, 27, 28, 29] or to directly output actions [30, 31, 32].”
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation (2019, background): “, 2018), or actual physical environments (Skočaj et al., 2016; Thomason et al., 2018; Williams et al., 2018).”
Grounded Language Learning: Where Robotics and NLP Meet (2018, result): “This approach is similar to research on learning object characteristics from exploratory robot interactions [Sinapov et al. , 2016; Thomason et al. , 2018] and, particularly, attempts to extend language models [She and Chai, 2016], but with the addition of learning more generalized concepts of…”