CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

By Oier Mees, Lukás Hermann, Erick Rosete-Beas, Wolfram Burgard. IEEE Robotics and Automation Letters, 2021

665 citations recorded by the source corpus.

General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse repertoire of general-purpose skills that allow composing long-horizon tasks by following unconstrained language instructions. In this letter, we present Composing Actions from Language and Vision (CALVIN) (Composing Actions from Language and Vision), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites. We evaluate the agents in zero-shot to novel language instructions and to novel environments. We show that a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents that learn to relate human language to their world models with this benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4be02694125b71876552900a53c85c47a2a83614

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

What Matters in Language Conditioned Robotic Imitation Learning Over Unstructured Data (2022, methodology): “recently proposed CALVIN benchmark [14] to further our understanding and provide a unified framework for longhorizon language conditioned policy learning.”
Code as Policies: Language Model Programs for Embodied Control (2022, methodology): “More recent methods learn the grounding end-to-end (language to action) [8]–[10], but they require copious amounts of training data, which can be expensive to obtain on real robots.”
Visual Language Maps for Robot Navigation (2022, background): “The recent success of large pretrained vision and language models [10], [27] has spurred a flurry of interest in applying their zero-sot capabilities to different domains including object detection and segmentation [28], [29], [11], robot manipulation [30], [31], [32], [33], and navigation [13], [12”
3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, positive): “For CALVIN, we compare with MCIL (Lynch & Sermanet, 2020), which is a conditional sequence-to-sequence variational autoencoder. matches the baseline performance in most tasks within the RLBench action prediction, showing its planning capability.”
Instruction-driven history-aware policies for robotic manipulations (2022, background): “Instruction-driven vision-based robotic manipulation has received growing attention for manipulations in 2D planar [57, 58] or recent 3D environments [8, 59, 60], and has been transferred to the real world [9, 10].”
RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation (2025, neutral): “…tasks for multi-task and meta-reinforcement learning, CALVIN pairs long-horizon, language-conditioned instructions with rich sensor suites [27], LIBERO [25] defines 130 lifelong-learning tasks with high-quality human teleoperation data, and RoboVerse [13] unifies multiple simulators and…”
Vision-Language Foundation Models as Effective Robot Imitators (2023, methodology): “To further validate the impact of model size on downstream robotic tasks, we train different variants with 10% of language annotated data in CALVIN, which is only 0.1% of the full data.”
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, neutral): “1: 3D Diffuser Actor sets a new state-of-the-art on RLBench [32] on a multi-view setup and on CALVIN [49] benchmarks on a zero-shot long-horizon setup.”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, neutral): “That is, if one policy performs better than another in real-world 1] , higher is better) for assessing the correlation between policy performances in real-world and simulation, as well as the overall quality of simulated evaluation pipelines.”
WorldSimBench: Towards Video Generation Models as World Simulators (2024, methodology): “RM We employ CALVIN (Mees et al., 2022) as the robot manipulation simulator, using only RGB images for the observation space and limiting the action space to the 7-DOF (degrees of freedom) of the robot arm.”
Octo: An Open-Source Generalist Robot Policy (2024, methodology): “This enables our model to learn control mostly from self-supervised visual observations and reduces the burden on language annotation, similar to prior work on multi-context imitation learning [54, 62, 61, 63].”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, methodology): “We follow the evaluation protocol in CALVIN (Mees et al., 2022c) and evaluate 1000 unique sequence instruction chains.”
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023, background): “However, a large number of robot learning works and benchmarks use quasi-static tasks [1,3–7,14,18,19,22,30,33,51, 55, 56, 64–66], making the system broadly applicable.”
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, methodology): “Prior work aimed to extract representations from play data for effective downstream policy learning [47] or learned self-supervised representations of skills, referred to as latent plans, using Conditional Variational Autoencoders (CVAE) [12, 21, 25, 24].”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “For benchmarking, several simulation environments exist [62, 24, 48], but we use RLBench [15] for its diversity of 6-DoF tasks and ease of generating expert demonstrations with templated language goals.”