Learning to Interpret Natural Language Commands through Human-Robot Dialog

By Jesse Thomason, Shiqi Zhang, Raymond J. Mooney, Peter Stone. 2015

200 citations recorded by the source corpus.

Subfields: Natural Language Processing, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In LLM Agents & Tool Use

Citations in the corpus

Grounded Language Learning in a Simulated 3D World (2017, background): “Other related work focuses on language grounding from the perspective of human-machine communication (Thomason et al., 2015; Wang et al., 2016; Arumugam et al., 2017).”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “Instruction following is a long-standing topic of interest in robotics [7, 10, 29, 34, 39, 40, 46, 51].”
A Roadmap Towards Machine Intelligence (2015, background): “The agents developed in this area can perform tasks, such as learning to execute instructions in natural environments by interacting with humans (Thomason et al., 2015), or improving performance on real-life video-games by consulting the instruction manual (Branavan et al.”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Several works have proposed systems for instructing robots with natural language [7, 8, 9, 10, 41, 42, 43, 44, 45, 46, 47].”
Instructable Intelligent Personal Agent (2016, background): “In recent work Thomason et al. (2015) use CCG parsing on natural language commands in order to execute them by a robot.”
ELLA: Exploration through Learned Language Abstraction (2021, background): “A long-standing goal for robotics and embodied agents is to build systems that can perform tasks speciﬁed in natural language [1, 8, 21, 26, 39, 42, 43].”
Learning Interpretable Spatial Operations in a Rich 3D Blocks World (2017, unknown): “For example, recent work (Thomason et al. 2015; 2016; 2017) has shown how these instructions can be combined with exploration on physical robotics to follow instructions and learn representations online.”
Inferring Rewards from Language in Context (2022, unknown): “…(MacMahon et al., 2006; Tellex et al., 2011; Chen and Mooney, 2011; Matuszek et al., 2013; Artzi and Zettlemoyer, 2013; She et al., 2014; Thomason et al., 2015; Wang et al., 2016; Fried et al., 2018a; Arumugam et al., 2017; Suhr et al., 2018) or map directly to primitive actions…”
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following (2020, background): “Natural language instruction following on physical robots is most commonly studied using handengineered symbolic representations of world state or instruction semantics [1, 7, 2, 3, 4, 8, 5, 9], which require representation design and state estimation pipelines that are hard to scale to complex envi”
Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight (2019, background): “Natural language instruction following has been extensively studied using hand-engineered symbolic intermediate representations of world state or instruction semantics with physical robots [1, 2, 3, 4, 17, 5, 18, 6, 19] and simulated agents [20, 21, 22, 23, 24, 25].”
Robots That Use Language (2020, methodology): “(88) used an approach that learned semantic parse information and grounded word meanings from dialogue interactions with users.”
An Application-Independent Approach to Building Task-Oriented Chatbots with Interactive Continual Learning (2020, methodology): “For robot navigation, [2, 39, 40, 16, 12] proposed methods for grounding navigation instructions.”
Executing Instructions in Situated Collaborative Interactions (2019, methodology): “Interactions were also used for semantic parser induction (Artzi and Zettlemoyer, 2011; Thomason et al., 2015; Wang et al., 2016).”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Several works have proposed methods for grounding language in robot actions [49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60].”
Verbalization: Narration of Autonomous Robot Experience (2016, methodology): “We next describe the use of our semantic annotations within our verbalization space.”
A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution (2021, background): “Natural language has been extensively studied in robotics research, including with focus on instruction [1, 24], reference resolution [25], question generation [26–28], and dialogue [4, 29, 30].”
Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog (2020, methodology): “For details on inducing utterance-denotation pairs from conversations, we refer to previous work [37].”
Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study (2019, background): “…semantic parsing has a wide range of applications, including question answering (Berant et al., 2013; Dong and Lapata, 2016; Finegan-Dollak et al., 2018), robot navigation (Artzi and Zettlemoyer, 2013; Thomason et al., 2015) and Web API calling (Quirk et al., 2015; Su et al., 2018).”