Robot Programming by Demonstration with situated spatial language understanding

By Maxwell Forbes, Rajesh P. N. Rao, Luke S. Zettlemoyer, Maya Cakmak. 2015

69 citations recorded by the source corpus.

Robot Programming by Demonstration (PbD) allows users to program a robot by demonstrating the desired behavior. Providing these demonstrations typically involves moving the robot through a sequence of states, often by physically manipulating it. This requires users to be co-located with the robot and have the physical ability to manipulate it. In this paper, we present a natural language based interface for PbD that removes these requirements and enables hands-free programming. We focus on programming object manipulation actions-our key insight is that such actions can be decomposed into known types of manipulator movements that are naturally described using spatial language; e.g., object reference expressions and prepositions. Our method takes a natural language command and the current world state to infer the intended movement command and its parametrization. We implement this method on a two-armed mobile manipulator and demonstrate the different types of manipulation actions that can be programmed with it. We compare it to a kinesthetic PbD interface and we demonstrate our method's ability to deal with incomplete language.

Subfields: Artificial Intelligence, Control and Systems Engineering, Mechanical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Mechanical Engineering

Citations in the corpus

Lifelong and Continual Learning Dialogue Systems: Learning during Conversation (2021, neutral): “Related research includes (Forbes et al. 2015; Wang et al. 2017).”
A formal semantics for situated conversation (2018, background): “Nor does the linguistic unit describe a concurrent nonlinguistic event, a focus of research on modelling multimodal demonstrations (Stojnic et al. 2013, Forbes et al. 2015).”
Robots That Use Language (2020, background): “A bottle consists of a bottle and a cap, and a person referencing “the bottle” may mean both, or they may say, “Grab the bottle and then turn the cap,”referring”
From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following (2019, methodology): “or the training tasks considered. Another related approach is semantic parsing, which has also been used to convert language into an executable form that corresponds to actions within an environment (Forbes et al., 2015; Misra et al., 2014; Tellex et al., 2011). In a related task to instruction foll”