Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models

By Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, Jonathan Tompson. 2022

84 citations recorded by the source corpus.

In recent years, much progress has been made in learning robotic manipulation policies that follow natural language instructions. Such methods typically learn from corpora of robot-language data that was either collected with specific tasks in mind or expensively re-labelled by humans with rich language descriptions in hindsight. Recently, large-scale pretrained vision-language models (VLMs) like CLIP or ViLD have been applied to robotics for learning representations and scene descriptors. Can these pretrained models serve as automatic labelers for robot data, effectively importing Internet-scale knowledge into existing datasets to make them useful even for tasks that are not reflected in their ground truth annotations? To accomplish this, we introduce Data-driven Instruction Augmentation for Language-conditioned control (DIAL): we utilize semi-supervised language labels leveraging the semantic understanding of CLIP to propagate knowledge onto large datasets of unlabelled demonstration data and then train language-conditioned policies on the augmented datasets. This method enables cheaper acquisition of useful language descriptions compared to expensive human labels, allowing for more efficient label coverage of large-scale datasets. We apply DIAL to a challenging real-world robotic manipulation domain where 96.5% of the 80,000 demonstrations do not contain crowd-sourced language annotations. DIAL enables imitation learning policies to acquire new capabilities and generalize to 60 novel instructions unseen in the original dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, LLM Agents & Tool Use, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In LLM Agents & Tool Use

In Reinforcement Learning

Citations in the corpus

Augmented Language Models: a Survey (2023, methodology): “RT-1 makes use of DIAL (Xiao et al., 2022), an approach for automatically labeling robot demonstrations with linguistic labels via the vision-language alignment model CLIP (Radford et al.”
PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “PaLM-E also outperforms the algorithm of Xiao et al. (2022) that leverages two CLIP models trained with hindsight relabeled data.”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “In DIAL [213], Data-driven Instruction Augmentation for Language-conditioned control is introduced.”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “To enable generalization in the wild, a parallel line of works has explored learning task specification from large-scale offline data [104–106, 35, 34, 44, 107, 108, 54], particularly egocentric videos [109, 110], or leveraging pre-trained foundation models [111–113, 33, 114, 115].”
TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation (2023, background): “Large Language Models (LLMs) have exhibited remarkable proficiency in generating text that closely resembles human language and in performing a wide range of tasks [69], including Natural Language Processing [4, 32, 54], Robotics [10, 43, 56], and Information Retrieval [1, 24, 25, 48].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “DIAL [215] focuses on augmenting language instructions in existing datasets using VLMs. RoboPoint [107] generates scenes procedurally with randomized 3D layouts, objects, and camera viewpoints.”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, methodology): “These visually-grounded models have been used for several downstream tasks, such as in robotic success detection [18, 20, 54, 65], action prediction [7, 56], and reward prediction [16, 22, 46, 48].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, methodology): “These prior approaches use VLMs for visual state representations [61], for identifying objects [49, 70], for high-level planning [17], or for providing supervision or success detection [73, 72, 74, 75, 76].”