3D-VLA: A 3D Vision-Language-Action Generative World Model

By Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, Chuang Gan. International Conference on Machine Learning, 2024

364 citations recorded by the source corpus.

Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from perception to action, neglecting the vast dynamics of the world and the relations between actions and dynamics. In contrast, human beings are endowed with world models that depict imagination about future scenarios to plan actions accordingly. To this end, we propose 3D-VLA by introducing a new family of embodied foundation models that seamlessly link 3D perception, reasoning, and action through a generative world model. Specifically, 3D-VLA is built on top of a 3D-based large language model (LLM), and a set of interaction tokens is introduced to engage with the embodied environment. Furthermore, to inject generation abilities into the model, we train a series of embodied diffusion models and align them into the LLM for predicting the goal images and point clouds. To train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by extracting vast 3D-related information from existing robotics datasets. Our experiments on held-in datasets demonstrate that 3D-VLA significantly improves the reasoning, multimodal generation, and planning capabilities in embodied environments, showcasing its potential in real-world applications.

Subfields: Automotive Engineering, Control and Systems Engineering, Geology

View Paper: https://www.semanticscholar.org/paper/e788d5c6dc79f106296f8ad296bcbe512028d5f0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Control and Systems Engineering

In Geology

Citations in the corpus

Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, positive): “…vision-language models (VLMs) to directly predict low-level robotic control actions, producing “vision-language-action” models (VLAs) [4, 33, 23, 22, 7, 15, 8, 50, 55, 51, 3, 2], which have demonstrated effective generalization to out-of-distribution test conditions and unseen semantic concepts.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “By leveraging this knowledge, we can improve model generalization, robustness, adaptability to new environments, and predictive accuracy on unseen data [200], [201].”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, background): “A number of recent works have explored similar recipes to ours and directly fine-tuned large pretrained VLMs for predicting robot actions [1, 7, 17, 18, 72–74].”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, neutral): “It serves as the backbone for various downstream tasks in robotics manipulation, such as learning viewpoint invariant representations [6, 56] or grounding actions through 3D vision-and-language models [52, 66], thus enabling the robotics agents to achieve geometric and visual generalization.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “3D-VLA [85] proposes a 3D world model capable of goal generation.”
π0.5: a Vision-Language-Action Model with Open-World Generalization (2025, positive): “Vision-language-action models (VLAs) [23, 92, 42, 8, 83, 90, 55, 45, 3, 75, 64, 76, 84, 7, 37] offer an appealing solution: by ﬁne-tuning pre-trained vision-language models for robot control, VLAs can leverage the semantic knowledge acquired from web-scale pretraining and bring it to bear on the…”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, neutral): “Recent advances in robot learning have demonstrated impressive progress in training policies that can act across diverse tasks and environments [1, 4, 10, 12, 15, 26, 33, 39, 40, 43, 48, 53, 57, 60, 64, 70, 72].”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, positive): “Moreover, 3D-VLA [69] builds a generative world model on top of 3D-based LLM to perform 3D reasoning and localization, multimodal goal generation, and embodied action planning.”