Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0

By A. Padalkar, A. Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, A. Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan and 164 more. IEEE International Conference on Robotics and Automation, 2023

1,105 citations recorded by the source corpus.

Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/ef7d31137ef06c5be8c2824ecc5af6ce3358cc8f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

Open-TeleVision: Teleoperation with Immersive Active Visual Feedback (2024, neutral): “If we distinguish the existing work with sources of demonstrations, we can classify them into learning from real-robot expert data [14, 10, 12, 59, 60, 61, 62, 50, 63, 1, 64, 51, 65], learning from play data [66, 67, 68], and learning from human demonstrations [67, 69, 70, 71, 72, 73, 74, 56].”
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, positive): “While Large Language Models (LLMs) [3, 6, 9, 36, 61, 67, 68, 77, 81, 82, 88, 105] have advanced linguistic intelligence, visual-spatial intelligence remains under-explored, despite its relevance to robotics [7, 8, 21, 64], autonomous driving [79], and AR/VR [12, 27, 55].”
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, positive): “Empowered by the emergence of large-scale robotic datasets [78, 63, 38, 1], robot policies based on vision-language-action models (VLA) have made encouraging strides recently [10, 28, 39].”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, neutral): “Given these challenges, a natural question arises: How can we build VLA models that retain the advantages of existing VLA models while being both fast and data-efﬁcient?”
Vision-Language Foundation Models as Effective Robot Imitators (2023, methodology): “This representative episode vividly illustrates that our method is much more effective and efficient and could better generalize to unseen vision context.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “This includes creating models that can do OCR [12, 220], visual grounding [12, 26, 143, 189, 207, 212, 219], image-text-retrieval [97], additional languages [112], embodied AI tasks [17, 135, 140, 152] or leverage other expert systems [52].”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “VLMs can be used in various scenarios, especially robotics [4, 5, 15, 23, 37].”
3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, positive): “The tasks include 1) embodied QA on RoboVQA dataset (Sermanet et al., 2023); 2) task captioning on 11 Open-X datasets (Padalkar et al., 2023), where we input the initial and final scenes and ask the agent to reason what has happened; 3) what-if QA on RT-1 dataset (Brohan et al., 2022), where the…”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, positive): “We use the Open X-Embodiment (OXE) [48] dataset as our primary training dataset.”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, positive): “Recent advances in robot learning have demonstrated impressive progress in training policies that can act across diverse tasks and environments [1, 4, 10, 12, 15, 26, 33, 39, 40, 43, 48, 53, 57, 60, 64, 70, 72].”
Foundation models in robotics: Applications, challenges, and the future (2023, positive): “RT-X [46] also investigates the possibility of training large cross-embodied robotic models in the domain of robotic manipulation.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “RT-X [143] builds upon the previous RT-1 and RT-2 models.”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, background): “…more modalities can take more flexible input signals so that people can interact with the model in more diverse manners, e.g., GPT-4o-like multi-modal chatbot, multi-modal web agent (Koh et al., 2024), and real-world robotics foundation model (Brohan et al., 2022; 2023; Padalkar et al., 2023).”
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, positive): “We use OpenVLA [22] as our representative base VLA, a 7B-parameter manipulation policy created by fine-tuning the Prismatic VLM [20] on 1M episodes from the Open X-Embodiment dataset [33].”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, positive): “More recent efforts [48, 39, 65] explore to use large-scale, cross-embodiment robot datasets [13] for generalist polices pre-training, supporting effective fine-tuning to new robot setups.”
OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning (2024, background): “Even one of the most comprehensive datasets to date, Open X-Embodiment [40], does not include data for humanoids.”