Learning from Unscripted Deictic Gesture and Language for Human-Robot Interactions

By Cynthia Matuszek, Liefeng Bo, Luke S. Zettlemoyer, Dieter Fox. 2014

158 citations recorded by the source corpus.

As robots become more ubiquitous, it is increasingly important for untrained users to be able to interact with them intuitively. In this work, we investigate how people refer to objects in the world during relatively unstructured communication with robots. We collect a corpus of deictic interactions from users describing objects, which we use to train language and gesture models that allow our robot to determine what objects are being indicated. We introduce a temporal extension to state-of-the-art hierarchical matching pursuit features to support gesture understanding, and demonstrate that combining multiple communication modalities more effectively captures user intent than relying on a single type of input. Finally, we present initial interactions with a robot that uses the learned models to follow commands while continuing to learn from user input.

Subfields: Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In LLM Agents & Tool Use

In Human-Computer Interaction

Citations in the corpus

LILA: Language-Informed Latent Actions (2021, methodology): “For example, Matuszek et al. [56] learn to map un-scripted interactions consisting of language and gestures to object localizations from relatively few interactions; such methods are crucial for scaling up LILA to multiple objects, referring expressions, and compositional language instructions.”
End-to-End Driving Via Conditional Imitation Learning (2017, background): “Human guidance of robot actions has been studied extensively [5], [14], [24], [34], [35].”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “While language-grounding for manipulation has been explored in the past [7, 8, 9, 10], these pipelines are limited by object-centric representations that cannot handle granular or deformable objects and often do not reason about perception and action in an integrated manner.”
Interpreting multimodal referring expressions in real time (2016, background): “This system combines the modalities of head direction and speech, but not any other form of gesture.”
Natural Language Communication with Robots (2016, background): “Progress on language in this area has largely focused on grounding visual attributes (Kollar et al., 2013; Matuszek et al., 2014) and on learning spatial relations and actions for small vocabularies with hard-coded abstract concepts (Steels and Vogt, 1997; Roy, 2002; Guadarrama et al., 2013).”
Situated Language Understanding with Human-like and Visualization-Based Transparency (2016, background): “Researchers have also incorporated perception of human gesture and gaze to enhance verbal language understanding [8, 20, 22, 31].”
Learning Interpretable Spatial Operations in a Rich 3D Blocks World (2017, background): “The intersection of robotics and language have seen impressive results in grounding visual attributes (Kollar, Krishnamurthy, and Strimel 2013; Matuszek et al. 2014), spatial reasoning (Steels and Vogt 1997; Roy 2002; Guadarrama et al. 2013), and action taking (MacMahon, Stankiewicz, and Kuipers…”
Towards a Dataset for Human Computer Communication via Grounded Language Acquisition (2016, background): “Progress on language has largely focused on grounding visual attributes (Kollar, Krishnamurthy, and Strimel 2013; Matuszek et al. 2014) and on learning spatial relations and actions for small vocabularies with hard-coded abstract concepts (Steels and Vogt 1997; Roy 2002; Guadarrama et al.”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Several works have proposed methods for grounding language in robot actions [49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60].”