Learning to Mediate Perceptual Differences in Situated Human-Robot Dialogue

By Changsong Liu, Joyce Yue Chai. 2015

21 citations recorded by the source corpus.

In human-robot dialogue, although a robot and its human partner are co-present in a shared environment, they have significantly mismatched perceptual capabilities (e.g., recognizing objects in the surroundings). When a shared perceptual basis is missing, it becomes difficult for the robot to identify referents in the physical world that are referred to by the human (i.e., a problem of referential grounding). To overcome this problem, we have developed an optimization based approach that allows the robot to detect and adapt to perceptual differences. Through online interaction with the human, the robot can learn a set of weights indicating how reliably/unreliably each dimension (e.g., object type, object color, etc.) of its perception of the environment maps to the human's linguistic descriptors and thus adjust its word models accordingly. Our empirical evaluation has shown that this weight-learning approach can successfully adjust the weights to reflect the robot's perceptual limitations. The learned weights, together with updated word models, can lead to a significant improvement for referential grounding in future dialogues.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Experience Grounds Language (2020, background): “resolved via dialog-based exploration with a human interlocutor (Liu and Chai, 2015), even as new object properties, like texture and weight (Thomason et al.”
Physical Causality of Action Verbs in Grounded Language Understanding (2016, background): “Although recent years have seen an increasing amount of work on grounding language to perception (Yu and Siskind, 2013; Walter et al., 2013; Liu et al., 2014; Naim et al., 2015; Liu and Chai, 2015), no previous work has investigated the link between physical causality denoted by action verbs and the”
Collaborative Language Grounding Toward Situated Human-Robot Dialogue (2017, methodology): “Gao, Q.; Doering, M.; Yang, S.; and Chai, J. Y. 2016.”
Situated Language Understanding with Human-like and Visualization-Based Transparency (2016, background): “Closely related to our work, Liu and Chai find that perceptual differences between humans and robots result in communication challenges [21].”