Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation

By Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, Chelsea Finn. 2021

189 citations recorded by the source corpus.

We study the problem of learning a range of vision-based manipulation tasks from a large offline dataset of robot interaction. In order to accomplish this, humans need easy and effective ways of specifying tasks to the robot. Goal images are one popular form of task specification, as they are already grounded in the robot's observation space. However, goal images also have a number of drawbacks: they are inconvenient for humans to provide, they can over-specify the desired behavior leading to a sparse reward signal, or under-specify task information in the case of non-goal reaching tasks. Natural language provides a convenient and flexible alternative for task specification, but comes with the challenge of grounding language in the robot's observation space. To scalably learn this grounding we propose to leverage offline robot datasets (including highly sub-optimal, autonomously collected data) with crowd-sourced natural language labels. With this data, we learn a simple classifier which predicts if a change in state completes a language instruction. This provides a language-conditioned reward function that can then be used for offline multi-task RL. In our experiments, we find that on language-conditioned manipulation tasks our approach outperforms both goal-image specifications and language conditioned imitation techniques by more than 25%, and is able to perform visuomotor tasks from natural language, such as"open the right drawer"and"move the stapler", on a Franka Emika Panda robot.

Subfields: Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “In BC NL we feed the full instruction i into the policy – this approach is representative of standard RL or BC-based instruction following methods [13, 20, 21, 22].”
PaLM-E: An Embodied Multimodal Language Model (2023, background): “While many works focus on understanding natural language goals (Lynch & Sermanet, 2020; Shridhar et al., 2022a; Nair et al., 2022; Lynch et al., 2022), fewer consider natural language as a representation for planning – the focus of this work.”
Language to Rewards for Robotic Skill Synthesis (2023, methodology): “Although these methods can achieve challenging language conditioned robotic tasks such as object pushing [25], and drawer opening [42], they require considerable language-labeled data to train the reward model.”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “Other works have incorporated pre-trained language models, often either as an instruction encoder [63, 64, 43, 41, 44, 1, 65] or for high-level planning [66, 19, 17, 67, 68, 69].”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Multimodal Large Language Models (MLLMs) have gained significant interest due to their proficiency in analyzing non-textual data like images and point clouds through text analysis [3, 139, 170, 203].”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, methodology): “For example, Nair et al. (181) learned a language-conditioned policy for table-top manipulation using model-based RL on a pre-collected dataset with hand-labeled language instructions.”
What Matters in Language Conditioned Robotic Imitation Learning Over Unstructured Data (2022, background): “[9] learn a reward classiﬁer which predicts if a change in state completes a language instruction and leverage it for ofﬂine multi-task RL given four camera views.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “[71] Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn.”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “Following previous works that leverage pre-trained language embed-dings [8, 40, 45, 103, 107–112] and pre-trained vision-language models [9, 113–115] in robotic imitation learning, we study both forms of pre-training in our experiments, specifically following the recipes of RT-1 [8] and RT-2 [9].”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “These approaches, as well as more modern research [8], [21], [22], [23], [24], [25], [26], [27], [28], typically require a large corpus of commands or observations with granular instructions, as well as a detailed alignment of commands to states.”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, background): “To enable generalization in the wild, a parallel line of works has explored learning task specification from large-scale offline data [104–106, 35, 34, 44, 107, 108, 54], particularly egocentric videos [109, 110], or leveraging pre-trained foundation models [111–113, 33, 114, 115].”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, methodology): “As in RT-1, several works use language commands processed with Transformers as a robust framework for specifying and generalizing to new tasks (Zhang & Chai, 2021; Pashevich et al., 2021; Silva et al., 2021; Jang et al., 2021; Ahn et al., 2022; Nair et al., 2022).”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, methodology): “These visually-grounded models have been used for several downstream tasks, such as in robotic success detection [18, 20, 54, 65], action prediction [7, 56], and reward prediction [16, 22, 46, 48].”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Some prior RL methods have also used contrastive learning to acquire reward functions [14, 20, 33, 38, 63, 67, 92, 133, 134, 146], often in imitation learning settings [37, 55].”
Code as Policies: Language Model Programs for Embodied Control (2022, unknown): “, model-based [33]–[35], imitation learning [8], [9], [36], [37], or reinforcement learning [38]–[42]).”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, background): “Recently, a number of end-to-end approaches [13, 12, 67, 68, 69] have been proposed for conditioning BC agents with language instructions.”