Back to the Blocks World: Learning New Actions through Situated Human-Robot Dialogue

By Lanbo She, Shaohua Yang, Yu Cheng, Yunyi Jia, Joyce Yue Chai, Ning Xi. 2014

97 citations recorded by the source corpus.

This paper describes an approach for a robotic arm to learn new actions through dialogue in a simplified blocks world. In particular, we have developed a threetier action knowledge representation that on one hand, supports the connection between symbolic representations of language and continuous sensorimotor representations of the robot; and on the other hand, supports the application of existing planning algorithms to address novel situations. Our empirical studies have shown that, based on this representation the robot was able to learn and execute basic actions in the blocks world. When a human is engaged in a dialogue to teach the robot new actions, step-by-step instructions lead to better learning performance compared to one-shot instructions.

Subfields: Artificial Intelligence, Control and Systems Engineering, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Experience Grounds Language (2020, background): “Real world, language-guided robots for task completion (Tellex et al., 2014) and learning (She et al., 2014) face challenging, continuous perception and control (Tellex et al., 2020).”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “On the other hand, experimentation on real humans and robots, despite expensive and time-consuming, are important for understanding the true complexity of real-world scenarios (Chai et al., 2018, 2016; Ryb-ski et al., 2007; Mohan and Laird, 2014; Liu et al., 2016; She et al., 2014).”
Physical Causality of Action Verbs in Grounded Language Understanding (2016, unknown): “To make such translation possible, previous works (She et al., 2014a; She et al., 2014b; Misra et al., 2015; She and Chai, 2016) explicitly model verbs with predicates describing the resulting states of actions.”
Collaborative Language Grounding Toward Situated Human-Robot Dialogue (2017, methodology): “For example, in our experiment with a SCHUNK robotic arm (She et al. 2014a), a teacher used 4 steps to teach the stack action.”
Language to Action: Towards Interactive Task Learning with Physical Agents (2018, methodology): “Therefore, another direction of our work is learning grounded verb semantics which explicitly models verb semantics (particularly for result verbs) as the desired goal states [She et al. , 2014; She and Chai, 2016].”
Inferring Rewards from Language in Context (2022, unknown): “…semantic representations (MacMahon et al., 2006; Tellex et al., 2011; Chen and Mooney, 2011; Matuszek et al., 2013; Artzi and Zettlemoyer, 2013; She et al., 2014; Thomason et al., 2015; Wang et al., 2016; Fried et al., 2018a; Arumugam et al., 2017; Suhr et al., 2018) or map directly to…”
Opportunistic Active Learning for Grounding Natural Language Descriptions (2017, background): “There has been some work on task-oriented dialog learning for robotics, but that work did not incorporate language grounding with perceptual predicates [18], did not involve much initiative from the robot [19], or aimed at eliciting human collaboration required only for completing the current task [”