Robots That Use Language

By Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, Cynthia Matuszek. 2020

312 citations recorded by the source corpus.

This article surveys the use of natural language in robotics from a robotics point of view. To use human language, robots must map words to aspects of the physical world, mediated by the robot's sensors and actuators. This problem differs from other natural language processing domains due to the need to ground the language to noisy percepts and physical actions. Here, we describe central aspects of language use by robots, including understanding natural language requests, using language to drive learning about the physical world, and engaging in collaborative dialogue with a human partner. We describe common approaches, roughly divided into learning methods, logic-based methods, and methods that focus on questions of human–robot interaction. Finally, we describe several application domains for language-using robots.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Experience Grounds Language (2020, background): “world (Tellex et al., 2011; Matuszek, 2018; Thomason et al., 2020; Tellex et al., 2020) must translate from language to action.”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “There is a long history of research studying how to connect language and behavior [46, 47, 48, 49, 50].”
LILA: Language-Informed Latent Actions (2021, background): “We additionally build off of work at the intersection of language and robotics [19, 20, 21].”
PaLM-E: An Embodied Multimodal Language Model (2023, background): “…on massive textual data may lead to representations that relate to our physical world, connecting those representations to real-world visual and physical sensor modalities is essential to solving a wider range of grounded real-world problems in computer vision and robotics (Tellex et al., 2020).”
Lifelong and Continual Learning Dialogue Systems: Learning during Conversation (2021, background): “For example, authors of (Vogel and Jurafsky 2010; Misra, Langford, and Artzi 2017; Tellex et al. 2020) have explored deep learning and reinforcement learning to ground NL commands directly into executable actions.”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “Natural language guidance of robots (Tellex et al., 2020) has been explored in contexts from furniture assembly (Tellex et al., 2011) to quadcoptor ﬂight control (Blukis et al., 2019).”
What Matters in Language Conditioned Robotic Imitation Learning Over Unstructured Data (2022, background): “Natural language processing has recently received much attention in the field of robotics [13], following the advances made towards learning groundings between vision and language [14], [15] and grounding behaviors in language [16].”
Correcting Robot Plans with Natural Language Feedback (2022, background): “We choose natural language input thanks to its efﬁciency, accessibility, ease-of-use, and direct conveyance of the user’s intent [39].”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “Our approach opens up opportunities in future work for more natural human-robot interactions [42] while providing a useful goal embedding in the process.”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “There exists comprehensive review works on this topic [104, 164].”
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards (2020, background): “A number of prior approaches have been proposed to use language to guide a learning agent [9].”
Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, background): “It is useful in a broad range of tasks, such as language-conditioned manipulation [9, 10, 11, 12, 13, 14, 15], visual language navigation [16, 17], and visual question answering [18, 19, 20, 21, 22, 23, 24].”
Language Grounding with 3D Objects (2021, background): “Using natural language to work with robot partners is a long-standing goal in robotics [14].”
Lifelong Learning Dialogue Systems: Chatbots that Self-Learn On the Job (2020, background): “For example, authors of (MacMahon, Stankiewicz, and Kuipers 2006; Branavan et al. 2009; Vogel and Jurafsky 2010; Misra, Langford, and Artzi 2017; Fried et al. 2018; Tellex et al. 2020) have explored deep learning and reinforcement learning to ground NL commands directly into executable actions.”
CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks (2021, background): “Natural language processing has recently received much attention in the field of robotics [8], following the advances made towards learning groundings between vision”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “Language grounding has been studied extensively both in terms of intelligent agents [19–22] and of robotics [23, 6, 24, 25, 5, 7, 26], where language can be used as a tool for compositional goal specification [5, 27–33], semantic anchor for training multimodal representation [12, 34, 35], or as an i”
ChatGPT for Robotics: Design Principles and Model Abilities (2023, background): “Classically, modeling human-robot interactions using language is challenging because it forces the user to operate within a rigid set of instructions [26], or requires mathematically complex algorithms to keep track of multiple probability distributions over actions and target objects [27], [28].”
Code as Policies: Language Model Programs for Embodied Control (2022, background): “Robots that use language need it to be grounded (or situated) to reference the physical world and bridge connections between words, percepts, and actions [4].”
An Application-Independent Approach to Building Task-Oriented Chatbots with Interactive Continual Learning (2020, methodology): “Existing methods for building NLIs either train an end-to-end model [4, 42, 27, 13, 38] to map a NL command directly into an executable action or learns a semantic parser (SP) to parse the command into an intermediate logical form which is then translated into an executable action [47, 2, 1, 49,…”