RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

By Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Krzysztof Marcin Choromanski, Tianli Ding, Danny Driess, Kumar Avinava Dubey, Chelsea Finn, Peter R. Florence and 36 more. Conference on Robot Learning, 2023

3,674 citations recorded by the source corpus.

We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web. To this end, we propose to co-fine-tune state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks, such as visual question answering. In contrast to other approaches, we propose a simple, general recipe to achieve this goal: in order to fit both natural language responses and robotic actions into the same format, we express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens. We refer to such category of models as vision-language-action models (VLA) and instantiate an example of such a model, which we call RT-2. Our extensive evaluation (6k evaluation trials) shows that our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training. This includes significantly improved generalization to novel objects, the ability to interpret commands not present in the robot training data (such as placing an object onto a particular number or icon), and the ability to perform rudimentary reasoning in response to user commands (such as picking up the smallest or largest object, or the one closest to another object). We further show that incorporating chain of thought reasoning allows RT-2 to perform multi-stage semantic reasoning, for example figuring out which object to pick up for use as an improvised hammer (a rock), or which type of drink is best suited for someone who is tired (an energy drink).

View Paper: https://www.semanticscholar.org/paper/38939304bb760473141c2aca0305e44fbe04e6e8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Agentic AI: Autonomous Intelligence for Complex Goals—A Comprehensive Survey (2025, unknown): “For example, in robotic tasks [88], a meta-learning agent raised in one environment (let’s say, in the house) can move to an alternative environment (the outside) and assume the knowledge of moving within the outside environment.”
GPT-Driver: Learning to Drive with GPT (2023, methodology): “We note that language modeling has been employed in other tasks of computer vision and robotics, such as object detection Chen et al. (2021); Xue et al. (2022); Wang et al. (2023) and robotic control Brohan et al. (2023).”
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, unknown): “…et al. [36] improve VLA efficiency through new action to-kenization schemes, using vector quantization or discrete co-sine transform-based compression to represent action chunks (sequences of actions) with fewer tokens than simple per-dimension binning (as used in RT-2 [4] and OpenVLA [22]).”
RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (2024, unknown): “H for metric calculation details). methods (Zhao et al., 2023; Brohan et al., 2023; Kim et al., 2024), while simultaneously requiring scalabilty for stable large-scale training on multimodal data (text, vision, actions).”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “These include risks related to privacy, safety, job displacement, bias in decision-making, and the impact on social norms and human relationships. i) Applications: Most current VLAs focus on household and industrial settings [2], which shape the available datasets and models.”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, background): “…their capabilities to generate content in various modalities, asserting their dominance in other domains like audio (Agostinelli et al., 2023), speech (Rubenstein et al., 2023), code generation (Li et al., 2023), medical applications (Singhal et al., 2023) and robotics (Zitkovich et al., 2023).”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, unknown): “RT-2 [8] tokenizes 7D actions into discrete tokens and uses the VLM PaLI-X [13] to predict them autoregressively like language tokens.”
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (2025, unknown): “Early results suggest promising gains in generalization capabilities (Black et al., 2024; Brohan et al., 2023).”
OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning (2024, background): “Recent efforts [40, 41, 42, 43, 44, 45, 46, 47] have focused on collecting robotic data using various embodiments for different tasks.”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, unknown): “…introduced in Section IV to create SIMPLER (Simulated Manipulation Policy Evaluation for Real Robot Setups), a suite of simulated evaluation environments for commonly used real robot evaluation setups: the Google Robot from the RT series of works [6, 5, 11] and the WidowX BridgeV2 setup [66].”
Vision-Language Foundation Models as Effective Robot Imitators (2023, methodology): “Motivated by PaLM-E, Brohan et al. (2023) further introduced RT-2, which is based on RT-1 but is adapted to use large vision-language backbones like PaLI-X (Chen et al., 2023) and PaLM-E (Driess et al., 2023), train the policy utilizing robot manipulation data both the web data.”
Open-TeleVision: Teleoperation with Immersive Active Visual Feedback (2024, unknown): “If we distinguish the existing work with sources of demonstrations, we can classify them into learning from real-robot expert data [14, 10, 12, 59, 60, 61, 62, 50, 63, 1, 64, 51, 65], learning from play data [66, 67, 68], and learning from human demonstrations [67, 69, 70, 71, 72, 73, 74, 56].”
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators (2023, unknown): “More recently, we have seen that system performance continues to increase with larger scale demonstration datasets and can even generalize to open vocabulary language instructions [3], [4], [5], [6].”
AgiBot World Colosseo: A Large-Scale Manipulation Platform for Scalable and Intelligent Embodied Systems (2025, unknown): “Another line of work lies in the use of large, end-to-end models trained on robot trajectories with robotics data scaling up [4], [24], [14], [25].”
RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation (2025, unknown): “…enabled robust zero-shot generalization: RT-1 [3] unifies vision, language and actions in a single transformer for real-time kitchen tasks; RT-2 [2] co-fine-tunes large vision–language models on web and robot data to unlock semantic planning and object reasoning; diffusion-based RDT-1B [26] and…”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “…Chen et al. [40] LLM-BasedCodeGeneration e.g. ProgPrompt [41], Code-as-Policy [42], ChatGPT-Robotics [43] Robot Transformers e.g. RT-1 [44], RT-2 [45], RT-X [46], PACT [47], Xiao et al. [48], Radosavovic et al. [49], LATTE [50] R e l e v a n t t o R obo ti c s Perception…”
An Embodied Generalist Agent in 3D World (2023, unknown): “In contrast to the finding that VL data benefits embodied acting tasks in VLA co-training (Brohan et al., 2023), our observation implies that embodied acting tasks may harm VL capabilities in turn.”
π0: A Vision-Language-Action Flow Model for General Robot Control (2024, unknown): “We then further train our model to incorporate robot actions, turning it into a vision-language-action (VLA) model [7].”