Situated Language Understanding with Human-like and Visualization-Based Transparency

By Leah Perlmutter, Eric Kernfeld, Maya Cakmak. 2016

31 citations recorded by the source corpus.

Communication with robots is challenging, partly due to their differences from humans and the consequent discrepancy in people’s mental model of what robots can see, hear, or understand. Transparency mechanisms aim to mitigate this challenge by providing users with information about the robot’s internal processes. While most research in human-robot interaction aim towards natural transparency using human-like verbal and non-verbal behaviors, our work advocates for the use of visualization-based transparency. In this paper, we first present an end-to-end system that infers task commands that refer to objects or surfaces in everyday human environments, using Bayesian inference to combine scene understanding, pointing detection, and speech recognition. We characterize capabilities of this system through systematic tests with a corpus collected from people (N=5). Then we design human-like and visualizationbased transparency mechanisms and evaluate them in a user study (N=20). The study demonstrates the effects of visualizations on the accuracy of people’s mental models, as well as their effectiveness and efficiency in communicating task commands.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Improving Robot Controller Transparency Through Autonomous Policy Explanation (2017, background): “Such control software can be complex to parse and interpret [21], particularly for modern robotics applications [20, 32].”
Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms (2018, unknown): “Instead of jointly learning an association between language and the full space of groundings, Tellex et al. (2011) and Liang et al.”
A survey of robot learning from demonstrations for Human-Robot Collaboration (2017, background): “Leah Perlmutter et al. tried to make robots provide their internal states to human users for helping them to understand the robot’s thought and intentions, since we as humans are not able to judge what robots can see, hear, or infer in the same way that we use in human-human communication [31].”