RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation

By Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, Jun Zhu. International Conference on Learning Representations, 2024

756 citations recorded by the source corpus.

Bimanual manipulation is essential in robotics, yet developing foundation models is extremely challenging due to the inherent complexity of coordinating two robot arms (leading to multi-modal action distributions) and the scarcity of training data. In this paper, we present the Robotics Diffusion Transformer (RDT), a pioneering diffusion foundation model for bimanual manipulation. RDT builds on diffusion models to effectively represent multi-modality, with innovative designs of a scalable Transformer to deal with the heterogeneity of multi-modal inputs and to capture the nonlinearity and high frequency of robotic data. To address data scarcity, we further introduce a Physically Interpretable Unified Action Space, which can unify the action representations of various robots while preserving the physical meanings of original actions, facilitating learning transferrable physical knowledge. With these designs, we managed to pre-train RDT on the largest collection of multi-robot datasets to date and scaled it up to 1.2B parameters, which is the largest diffusion-based foundation model for robotic manipulation. We finally fine-tuned RDT on a self-created multi-task bimanual dataset with over 6K+ episodes to refine its manipulation capabilities. Experiments on real robots demonstrate that RDT significantly outperforms existing methods. It exhibits zero-shot generalization to unseen objects and scenes, understands and follows language instructions, learns new skills with just 1~5 demonstrations, and effectively handles complex, dexterous tasks. We refer to https://rdt-robotics.github.io/rdt-robotics/ for the code and videos.

Subfields: Biomedical Engineering, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/41ba8127b73abd32051a89764cb976095b1dee94

Citations in the corpus

• RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation (2025, neutral): “…and actions in a single transformer for real-time kitchen tasks; RT-2 [2] co-fine-tunes large vision–language models on web and robot data to unlock semantic planning and object reasoning; diffusion-based RDT-1B [26] and the π 0 [1] capture diverse bimanual dynamics from over a million episodes.”
• AgiBot World Colosseo: A Large-Scale Manipulation Platform for Scalable and Intelligent Embodied Systems (2025, neutral): “Through unified pre-training on web-scale data, spanning human videos to our high-quality robot dataset, GO-1 achieves superior generalization and dexterity, outperforming prior generalist policies such as RDT [10] and our variant without latent action planner.”
• SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, neutral): “More recent efforts [48, 39, 65] explore to use large-scale, cross-embodiment robot datasets [13] for generalist polices pre-training, supporting effective fine-tuning to new robot setups.”
• CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, neutral): “For example, [24] trained a diffusion trans-former with 221M parameters and [38] further scaled the action model size to 1B.”
• A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “Many have made significant progress [139], [147], but still lacks generalization capability on par with LLMs in NLP. Attaining [231] and LanguageBind [232], align different modalities to the image or language embedding space, respectively.”
• Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, neutral): “Prior works [51, 26, 3] have shown that vanilla LoRA fine-tuning with autoregressive VLAs [22] is impractical for such tasks, as its throughput (3-5 Hz for single-arm robots and even lower for bimanual tasks) falls well below the 25-50 Hz required for real-time deployment.”