Interactive Language: Talking to Robots in Real Time

By Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, Peter R. Florence. 2022

343 citations recorded by the source corpus.

We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuo-linguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g."make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Robotics & Control

Citations in the corpus

PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “Prior works focus on combining vision and language inputs in an embodied setting with the goal of direct action prediction (Guhur et al., 2022; Shridhar et al., 2022b;a; Zhang & Chai, 2021; Silva et al., 2021; Jang et al., 2022; Nair et al., 2022; Lynch et al., 2022; Brohan et al., 2022).”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, methodology): “…Play [95] 0.4% Berkeley Cable Routing [96] 0.2% Roboturk [97] 2.3% Viola [98] 0.9% Berkeley Autolab UR5 [99] 1.2% Toto [100] 2.0% Language Table [101] 4.4% Stanford Hydra Dataset [102] 4.4% Austin Buds Dataset [103] 0.2% NYU Franka Play Dataset [104] 0.8% Furniture Bench Dataset [105] 2.4% UCSD…”
3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, unknown): “…here, with details available in the Appendix: Robot Datasets: We select 12 datasets (Brohan et al., 2022; Jang et al., 2022; Walke et al., 2023; Lynch et al., 2023; Feng et al., 2023; Chen et al., 2023a; Dass et al., 2023; Mandlekar et al., 2019; Mees et al., 2023; Shah et al., 2023; Sawhney…”
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, unknown): “…Play [21] 0.6% Berkeley Cable Routing [51] 0.3% Roboturk [55] 2.8% Viola [99] 1.1% Berkeley Autolab UR5 [15] 1.4% Toto [95] 2.4% Language Table [53] 5.2% Stanford Hydra Dataset [7] 5.3% Austin Buds Dataset [98] 0.3% NYU Franka Play Dataset [19] 1.0% Furniture Bench Dataset [31] 2.9% UCSD…”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Interactive Language [100] presents a robotic system wherein the low-level control policy can be guided in real-time by human instructions conveyed through language, enabling the completion of long-horizon rearrangement tasks.”
Octo: An Open-Source Generalist Robot Policy (2024, methodology): “Fractal [10] 17.0% Kuka [41] 17.0% Bridge[25, 90] 17.0% BC-Z [39] 9.1% Stanford Hydra Dataset [5] 6.0% Language Table [55] 5.9% Taco Play [77, 63] 3.6% Furniture Bench Dataset [33] 3.3% UTAustin Mutex [82] 3.0% Austin Sailor Dataset [65] 2.9% Roboturk [58] 2.8% Toto [99] 2.4% Austin Sirius Dataset…”
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, background): “However the performance of such methods is critically dependent on the quantity and breadth of instruction-labeled demonstration data that is available [32], and producing expert demonstrations of robot motion often requires expertise and time [33].”
MIMIC-IT: Multi-Modal In-Context Instruction Tuning (2023, methodology): “In the future, we plan to support more embodied AI datasets such as Language-Table [29] and SayCan [1].”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “Even the largest data collection efforts still end up with datasets that are a fraction of the size and diversity of benchmark datasets in vision (5-18M) [4, 5] and NLP (1.5B-4.”
The rise and potential of large language model based agents: a survey (2025, unknown): “The physical environment provides rich sensory inputs, including visual [101,270], auditory [265,268], and spatial senses.”
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023, background): “One popular data source is human demonstrators that teleoperate robot arms [2–6, 23– 27 ], but collecting 3 Problem Setup Imitation Learning.”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, unknown): “2022; Ahn et al., 2022; Driess et al., 2023; Du et al., 2023; Guhur et al., 2023; Jang et al., 2022; Lynch & Sermanet, 2020; Lynch et al., 2022; Mees et al., 2022a;c; Ha et al., 2023).”
Language to Rewards for Robotic Skill Synthesis (2023, methodology): “To extend the end-to-end approaches to manipulation, a common approach is to utilize latent embeddings of language commands as multitask input context, and train with behavioral cloning [14, 32, 16], offline reinforcement learning [33], goal-conditioned reinforcement learning [34], or in a shared au”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, unknown): “…BC-Z [26] 8.64% 43264 6015535 Kuka [27] 7.06% 209880 2455879 RH20T [18] 5.67% 104392 52644433 Stanford Hydra [2] 5.15% 570 358234 Language Table [43] 5.06% 442226 7045476 Taco Play [56, 45] 3.42% 3603 237798 Furniture Bench [23] 2.84% 5100 3948057 Roboturk [44] 2.69% 1995 187507 Utaustin Mutex…”