Language Conditioned Imitation Learning Over Unstructured Data

By Corey Lynch, Pierre Sermanet. 2020

303 citations recorded by the source corpus.

Natural language is perhaps the most flexible and intuitive way for humans to communicate tasks to a robot. Prior work in imitation learning typically requires each task be specified with a task id or goal image -- something that is often impractical in open-world environments. On the other hand, previous approaches in instruction following allow agent behavior to be guided by language, but typically assume structure in the observations, actuators, or language that limit their applicability to complex settings like robotics. In this work, we present a method for incorporating free-form natural language conditioning into imitation learning. Our approach learns perception from pixels, natural language understanding, and multitask continuous control end-to-end as a single neural network. Unlike prior work in imitation learning, our method is able to incorporate unlabeled and unstructured demonstration data (i.e. no task or language labels). We show this dramatically improves language conditioned performance, while reducing the cost of language annotation to less than 1% of total data. At test time, a single language conditioned visuomotor policy trained with our method can perform a wide variety of robotic manipulation skills in a 3D environment, specified only with natural language descriptions of each task (e.g."open the drawer...now pick up the block...now press the green button..."). To scale up the number of instructions an agent can follow, we propose combining text conditioned policies with large pretrained neural language models. We find this allows a policy to be robust to many out-of-distribution synonym instructions, without requiring new demonstrations. See videos of a human typing live text commands to our agent at language-play.github.io

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Various works have investigated strategies for the challenging problem of fusing vision, language, and control [60, 61, 62, 63, 64, 65, 66].”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “At the same time, by leveraging the better representation power of neural architectures, a number of works have looked into creating instruction-following agents that can perform manipulation [29, 30], navigation [11, 54, 31], or both [49, 16, 12].”
PaLM-E: An Embodied Multimodal Language Model (2023, background): “While many works focus on understanding natural language goals (Lynch & Sermanet, 2020; Shridhar et al., 2022a; Nair et al., 2022; Lynch et al., 2022), fewer consider natural language as a representation for planning – the focus of this work.”
Can Wikipedia Help Offline Reinforcement Learning? (2022, methodology): “Our work is orthogonal to these as we essentially amortize RL al-gorithm itself, expressed as sequence modeling with Trans-former, instead of speciﬁc RL domain information, and can be combined with domain-speciﬁc pre-training techniques (Yen-Chen et al., 2020; Lynch & Sermanet, 2021) effortlessly.”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “In a follow-up work [28], the authors present multi-context imitation (MCIL) which uses language-conditioned imitation learning over unstructured data.”
3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, unknown): “For CALVIN, we compare with MCIL (Lynch & Sermanet, 2020), which is a conditional sequence-to-sequence variational autoencoder. matches the baseline performance in most tasks within the RLBench action prediction, showing its planning capability.”
Learning Universal Policies via Text-Guided Video Generation (2023, background): “There has been a growing amount of work using text commands as a way to learn multi-task and generalist control policies (Huang et al., 2022; Ahn et al., 2022; Brohan et al., 2022; Lynch & Sermanet, 2020; Mahmoudieh et al., 2022; Liu et al., 2022).”
Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, background): “It is useful in a broad range of tasks, such as language-conditioned manipulation [9, 10, 11, 12, 13, 14, 15], visual language navigation [16, 17], and visual question answering [18, 19, 20, 21, 22, 23, 24].”
Vision-Language Foundation Models as Effective Robot Imitators (2023, methodology): “We compare a set of well-performed baselines in CALVIN: (1) MCIL (Lynch & Sermanet, 2020): a scalable framework combining multitask imitation with free-form text conditioning, which learns language-conditioned visuomotor policies, and is capable of following multiple human instructions over a long…”
Octo: An Open-Source Generalist Robot Policy (2024, methodology): “This enables our model to learn control mostly from self-supervised visual observations and reduces the burden on language annotation, similar to prior work on multi-context imitation learning [54, 62, 61, 63].”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “These approaches, as well as more modern research [8], [21], [22], [23], [24], [25], [26], [27], [28], typically require a large corpus of commands or observations with granular instructions, as well as a detailed alignment of commands to states.”
Code as Policies: Language Model Programs for Embodied Control (2022, methodology): “, model-based [33]–[35], imitation learning [8], [9], [36], [37], or reinforcement learning [38]–[42]).”
Learning Interactive Real-World Simulators (2023, methodology): “Base [33] dataset consists of 160k simulated trajectories and 440k real trajectories where each trajectory contains a language instruction (e.g., “move blue cube to the right”), a sequence of visuomotor controls, and a sequence of image frames corresponding to the execution of the task.”
Interactive Language: Talking to Robots in Real Time (2022, methodology): “The purpose of Dcollect is to provide a sufficiently diverse basis for crowdsourced hindsight language relabeling [8], [11], described next.”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “These advancements have greatly improved zero-shot and few-shot image classification [130, 139], segmentation [79, 103], and object detection [115].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “…if we augment the command with chain of thought prompting, the model is able to make even more complex semantic inferences, such as figuring out which object to pick up for use as an improvised hammer (a rock), or which type of drink is best suited for someone who is tired (an energy drink).”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, methodology): “We compare with four baseline methods: MCIL (Lynch & Sermanet, 2020), RT-1 (Brohan et al., 2022), HULC (Mees et al., 2022b), and a multi-task version of R3M (Nair et al., 2022).”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, background): “to address robotic language understanding through pipelined approaches that combine language parsing, vision, and robotic control [40, 31, 62] and with end-to-end approaches [41, 60, 39, 1].”