RT-1: Robotics Transformer for Real-World Control at Scale

By Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu and 41 more. 2022

2,545 citations recorded by the source corpus.

By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks. The project's website and videos can be found at robotics-transformer1.github.io

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Model Architectures

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “Finally, RT-1 (Brohan et al., 2022) leverages large-scale, diverse, task-agnostic robotic datasets to learn a model that can follow over 700 natural language instructions, as well as generalize to new tasks, environments, and objects.”
PaLM-E: An Embodied Multimodal Language Model (2023, background): “The low-level policies are from RT-1 (Brohan et al., 2022), a transformer model that takes RGB image and natural language instruction, and outputs end-effector control commands.”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, background): “Details of the scenario distribution are shown in Section A7.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “While discrete action has demonstrated superior performance in RT-1 [114], Octo [134] argues that it leads to early grasping issues.”
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (2024, methodology): “Previous approaches utilized interfaces such as 3D spacemouse [9, 54], VR or AR controllers [35, 3, 13, 19, 31, 51, 12], smartphones [44, 45, 22], and haptic devices [38, 47, 43, 26, 4] for teleoperation.”
BridgeData V2: A Dataset for Robot Learning at Scale (2023, methodology): “…with broader coverage and may benefit from noisy suboptimal trajectories [46], methods that require language descriptions to run language-conditioned imitation learning [5], or methods that use high-capacity models and require very large datasets before they can perform at their full potential [7].”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, positive): “Remarkably, our model even outperforms RT-1 [7], which is trained on a Google robot-specific dataset, by an average success rate margin of 22.4% in Visual Matching and 17.6% in Variant Aggregation .”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, neutral): “We analyze DROID along these axes and compare it to existing large-scale robot manipulation datasets [2, 14, 59].”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “Recent approaches exploring foundation models for robotics have demonstrated impressive open-set interaction abilities [36, 37, 38, 39, 40].”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “This method is broadly representative of prior language-conditioned behavioral cloning meth-ods such as RT-1 [6] and BC-Z [30].”
RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (2024, positive): “Such continuous encoding can avoid precision loss in contrast to discretized encoding (Brohan et al., 2022; 2023; Kim et al., 2024).”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, positive): “Instead, our aim is to give practitioners a readily available signal for policy improvement to guide their research.”
Open-TeleVision: Teleoperation with Immersive Active Visual Feedback (2024, positive): “If we distinguish the existing work with sources of demonstrations, we can classify them into learning from real-robot expert data [14, 10, 12, 59, 60, 61, 62, 50, 63, 1, 64, 51, 65], learning from play data [66, 67, 68], and learning from human demonstrations [67, 69, 70, 71, 72, 73, 74, 56].”
AppAgent: Multimodal Agents as Smartphone Users (2023, positive): “The emergence of multimodal LLM agents [3, 4, 11, 29, 36, 50], capable of processing various inputs including text, images, audio, and video, has further broadened the scope of LLM applications.”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, positive): “Generalist robot policies that are capable of interacting with the physical environment, adapting to various embodiments, and performing complex tasks have been a long-standing pursuit in robotics [6, 3, 16, 8, 65].”
RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation (2025, positive): “In contrast, foundation models trained on million-scale, multi-robot corpora have enabled robust zero-shot generalization: RT-1 [3] unifies vision, language and actions in a single transformer for real-time kitchen tasks; RT-2 [2] co-fine-tunes large vision–language models on web and robot data to…”
RVT: Robotic View Transformer for 3D Object Manipulation (2023, background): “With the recent breakthrough in large language models, multi-task robot learning has been approached by using natural language to specify a broad range of tasks and learning the policy from large pre-collected datasets [35, 26, 36, 2, 37, 38, 39, 40, 41].”
π0.5: a Vision-Language-Action Model with Open-World Generalization (2025, neutral): “…from narrow, single-task datasets to diverse datasets that span many scenes and tasks [17, 25, 80, 63, 41, 6, 30, 67, 1] allows the resulting policies to not only solve a wider range of tasks out of the box, but also improves their ability to generalize to new scenes and tasks [9, 63, 62, 22].”
OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning (2024, background): “Recent efforts [40, 41, 42, 43, 44, 45, 46, 47] have focused on collecting robotic data using various embodiments for different tasks.”