Asking for Help Using Inverse Semantics

By Stefanie Tellex, Ross A. Knepper, Adrian Shuai Li, Daniela Rus, Nicholas A. Roy. 2014

221 citations recorded by the source corpus.

Robots inevitably fail, often without the ability to recover autonomously. We demonstrate an approach for enabling a robot to recover from failures by communicating its need for specific help to a human partner using natural language. Our approach automatically detects failures, then generates targeted spoken-language requests for help such as “Please give me the white table leg that is on the black table.” Once the human partner has repaired the failure condition, the system resumes full autonomy. We present a novel inverse semantics algorithm for generating effective help requests. In contrast to forward semantic models that interpret natural language in terms of robot actions and perception, our inverse semantics algorithm generates requests by emulating the human’s ability to interpret a request using the Generalized Grounding Graph (G) framework. To assess the effectiveness of our approach, we present a corpusbased online evaluation, as well as an end-to-end user study, demonstrating that our approach increases the effectiveness of human interventions compared to static requests for help.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Experience Grounds Language (2020, background): “Real world, language-guided robots for task completion (Tellex et al., 2014) and learning (She et al.”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Ideas such as combining robot planning and language [85], using language models as a pre-training mechanism for policies [44] and many other ways of combining language and interaction [21, 22, 38, 86] are exciting avenues for future research.”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Some works utilize language for planning and grounding [39, 40, 41, 42, 43, 44].”
Improving Robot Controller Transparency Through Autonomous Policy Explanation (2017, background): “[42] investigated scenarios in which a human was relied upon to debug a robot’s behaviors during live task execution, and contributed an algorithm enabling the robot to”
Learning Language Games through Interaction (2016, background): “Examples include playing games (Branavan et al., 2009, 2010; Reckman et al., 2010) interacting with robotics (Tellex et al., 2011, 2014), and following instructions (Vogel and Jurafsky, 2010; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013) Semantic parsing utterances to logical forms, which we…”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “There has been a line of work for building agents that can execute natural language instructions: Tellex et al. (2011; 2014) for robotics and MacMa-hon et al. (2006); Chen and Mooney (2011); Mei et al. (2015) for a simulated environment.”
Vision-and-Dialog Navigation (2019, background): “g, moving a table leg to be within reach for the robot) [1].”
Learning Multi-Modal Grounded Linguistic Semantics by Playing "I Spy" (2016, methodology): “Incorporating a language model also allows for more robust generation of robot referring expressions for objects, as explored in [Tellex et al., 2014].”
Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog (2020, background): “One framework, based on generalized grounding graphs, aims 1 The demonstration video can be viewed at https://youtu.be/PbOfteZ CJc to both understand human language requests about objects in the world and generate language requests regarding the shared environment [36].”
Opportunistic Active Learning for Grounding Natural Language Descriptions (2017, background): “There has been some work on task-oriented dialog learning for robotics, but that work did not incorporate language grounding with perceptual predicates [18], did not involve much initiative from the robot [19], or aimed at eliciting human collaboration required only for completing the current task [”
Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding (2017, methodology): “Tellex et al. (2014) apply a similar technique, under the name inverse semantics , to create a robot that can informatively ask humans for assistance in accomplishing tasks.”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, background): “Since Simple Set does not provide calibrated coverage, we first find ϵ =0 .”
Interactive Language: Talking to Robots in Real Time (2022, background): “Our work exists in a larger setting of humans modifying or correcting the behavior of autonomous agents [31], historically addressed”
Robots That Use Language (2020, background): “This work has been extended to enable robots to ask natural language questions that clarify ambiguous commands (78, 93) and to enable robots to ask for help (94).”
A survey of robot learning from demonstrations for Human-Robot Collaboration (2017, background): “Stefanie Tellex et al. presented an approach for a robot to take advantage of receiving help from its human partner when the robot and the human partner work together for accomplishing a certain task [32].”
Correcting Robot Plans with Natural Language Feedback (2022, background): “1) Natural Language for Robot Behavior Correction: Correcting robot behavior using language has been studied for robots that ask for help [38], understand language corrections [10, 12], and use language to disambiguate joy-stick corrections [21].”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “Language grounding has been studied extensively both in terms of intelligent agents [19–22] and of robotics [23, 6, 24, 25, 5, 7, 26], where language can be used as a tool for compositional goal specification [5, 27–33], semantic anchor for training multimodal representation [12, 34, 35], or as an i”
Robobarista: Object Part Based Transfer of Manipulation Trajectories from Crowd-Sourcing in 3D Pointclouds (2015, background): “Among previous efforts to scale teaching to the crowd [9, 52, 21], Forbes”