Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularities

By Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, Lawson L. S. Wong, Stefanie Tellex. 2017

64 citations recorded by the source corpus.

Humans can ground natural language commands to tasks at both abstract and fine-grained levels of specificity. For instance, a human forklift operator can be instructed to perform a high-level action, like "grab a pallet" or a low-level action like "tilt back a little bit." While robots are also capable of grounding language commands to tasks, previous methods implicitly assume that all commands and tasks reside at a single, fixed level of abstraction. Additionally, methods that do not use multiple levels of abstraction encounter inefficient planning and execution times as they solve tasks at a single level of abstraction with large, intractable state-action spaces closely resembling real world complexity. In this work, by grounding commands to all the tasks or subtasks available in a hierarchical planning framework, we arrive at a model capable of interpreting language at multiple levels of specificity ranging from coarse to more granular. We show that the accuracy of the grounding procedure is improved when simultaneously inferring the degree of abstraction in language used to communicate the task. Leveraging hierarchy also improves efficiency: our proposed approach enables a robot to respond to a command within one second on 90% of our tasks, while baselines take over twenty seconds on half the tasks. Finally, we demonstrate that a real, physical robot can ground commands at multiple levels of abstraction allowing it to efficiently plan different subtasks within the same planning hierarchy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Grounded Language Learning in a Simulated 3D World (2017, background): “Other related work focuses on language grounding from the perspective of human-machine communication (Thomason et al., 2015; Wang et al., 2016; Arumugam et al., 2017).”
LILA: Language-Informed Latent Actions (2021, background): “similar to prior work in language-conditional robotics [10, 34, 42].”
Using Natural Language for Reward Shaping in Reinforcement Learning (2019, background): “[Williams et al. , 2017] and [Arumugam et al. , 2017] map language to a reward function in an object-oriented MDP framework.”
ELLA: Exploration through Learned Language Abstraction (2021, background): “This makes it difficult to use explicitly modular approaches (Arumugam et al., 2017; Andreas et al., 2017; Das et al., 2018) out of the box; instead need to be able to explore using a mixture of these low-level instructions as well as primitive actions.”
A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions (2017, methodology): “Additionally, Arumugam et al. (2017) simplify the problem from one of machine translation to multi-class classiﬁcation, learning a deep neural network to map arbitrary natural language instructions to the corresponding reward function.”
Robots That Use Language (2020, background): “Childhood education is a significant area of research for HRI studies (132), both because there is a chronic shortage of personnel in education and child care and because increasing the role population into STEM fields.”
Grounded Language Learning: Where Robotics and NLP Meet (2018, methodology): “, 2017], cognitive approaches [Mohan and Laird, 2014], and approaches in which meaning is represented as part of a larger-scale knowledge framework [Williams and Scheutz, 2016] or graphical structure [Arumugam et al., 2017].”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “Many prior works have studied this problem in the context of instruction following, where an agent aims to complete a task specified by formal language/programs [13, 14, 15, 16, 17, 18] or natural language [10, 11, 19, 20, 21, 22].”
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards (2020, background): “Arumugam et al. [11] propose using language to generate rewards at multiple levels of abstraction, by directly learning a conditional distribution of the level of abstraction and the reward function given a command.”
Language to Action: Towards Interactive Task Learning with Physical Agents (2018, methodology): “Recent work has applied deep learning models to directly map language instructions and raw visual observations to actions [Misra and Langford, 2017] or action representations [Arumugam et al. , 2017].”
Correcting Robot Plans with Natural Language Feedback (2022, background): “Mapping language to symbolic plans [37, 22, 24, 4], has enabled following instructions by invoking a set of pre-specified skills or motion primitives.”
Prospection: Interpretable plans from language by predicting the future (2019, background): “While we maintain an interpretable intermediate layer, our interface is natural language, most akin to [9], [10] though we work in”
Learning Adaptive Language Interfaces through Decomposition (2020, background): “One such communication modality that is both user-friendly and versatile is natural language; to this end, we focus on robust natural language interfaces (NLIs) that can map utterances to executable behavior (Tellex et al., 2011; Artzi and Zettlemoyer, 2013; Thomason et al., 2015; Arumugam et al., 2”
Inferring Rewards from Language in Context (2022, background): “…Mooney, 2011; Matuszek et al., 2013; Artzi and Zettlemoyer, 2013; She et al., 2014; Thomason et al., 2015; Wang et al., 2016; Fried et al., 2018a; Arumugam et al., 2017; Suhr et al., 2018) or map directly to primitive actions (Branavan et al., 2009; Andreas and Klein, 2015; Mei et al., 2016;…”