A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions

By Siddharth Karamcheti, Edward C. Williams, Dilip Arumugam, Mina Rhee, Nakul Gopalan, Lawson L. S. Wong, Stefanie Tellex. 2017

17 citations recorded by the source corpus.

Robots operating alongside humans in diverse, stochastic environments must be able to accurately interpret natural language commands. These instructions often fall into one of two categories: those that specify a goal condition or target state, and those that specify explicit actions, or how to perform a given task. Recent approaches have used reward functions as a semantic representation of goal-based commands, which allows for the use of a state-of-the-art planner to find a policy for the given task. However, these reward functions cannot be directly used to represent action-oriented commands. We introduce a new hybrid approach, the Deep Recurrent Action-Goal Grounding Network (DRAGGN), for task grounding and execution that handles natural language from either category as input, and generalizes to unseen environments. Our robot-simulation results demonstrate that a system successfully interpreting both goal-oriented and action-oriented task specifications brings us closer to robust natural language understanding for human-robot interaction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

LILA: Language-Informed Latent Actions (2021, background): “A variety of methods have sought to combine language and robotics, spanning approaches that map language to planning primitives [9, 30, 31, 32, 33], perform imitation learning from demonstrations and instructions [11, 34, 35, 36], and pair language instructions with reward functions for reinforcemen”
Learning Adaptive Language Interfaces through Decomposition (2020, background): “tions of reward functions (Arumugam et al., 2017; Karamcheti et al., 2017), or learning languageconditioned policies via reinforcement learning (Hermann et al.”
Zero-shot Learning of Classifiers from Natural Language Quantification (2018, background): “More generally, learning from language has also been previously explored in tasks such as playing games (Branavan et al., 2012), robot navigation (Karamcheti et al., 2017), etc. Natural language quantiﬁcation has been studied from multiple perspectives in formal logic (Barwise and Cooper, 1981),…”
Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback (2020, background): “, 2012), robot navigation (Karamcheti et al., 2017), concept learning (e.”
ELLA: Exploration through Learned Language Abstraction (2021, background): “Other hierarchical language techniques include learning models that predict symbolic representations of reward functions at different granularities (Arumugam et al., 2017; Karamcheti et al., 2017), and compositional approaches like policy sketches (Andreas et al.”