An Embodied Generalist Agent in 3D World

By Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, Siyuan Huang. International Conference on Machine Learning, 2023

406 citations recorded by the source corpus.

Leveraging massive knowledge from large language models (LLMs), recent machine learning models show notable successes in general-purpose task solving in diverse domains such as computer vision and robotics. However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e.g., 3D grounding, embodied reasoning and acting. We argue these limitations significantly hinder current models from performing real-world tasks and approaching general intelligence. To this end, we introduce LEO, an embodied multi-modal generalist agent that excels in perceiving, grounding, reasoning, planning, and acting in the 3D world. LEO is trained with a unified task interface, model architecture, and objective in two stages: (i) 3D vision-language (VL) alignment and (ii) 3D vision-language-action (VLA) instruction tuning. We collect large-scale datasets comprising diverse object-level and scene-level tasks, which require considerable understanding of and interaction with the 3D world. Moreover, we meticulously design an LLM-assisted pipeline to produce high-quality 3D VL data. Through extensive experiments, we demonstrate LEO's remarkable proficiency across a wide spectrum of tasks, including 3D captioning, question answering, embodied reasoning, navigation and manipulation. Our ablative studies and scaling analyses further provide valuable insights for developing future embodied generalist agents. Code and data are available on project page.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/13d12b26db345f62e8e512db181b96a7f8763b47

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “Real : SayCan setup LEO [159] ConvNeXt [160], PointNet++ [161] Vicuna-7B Concat MLP Yes Sim-Mani : CLIPort task subset; Sim-Navi : Habitat-web 3D-LLM [57] CLIP, 3D-CLR [162], ConceptFusion [163] Flamingo, BLIP-2”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, positive): “Some researches [73, 10, 19, 24, 53, 25, 69] have focused on extending the generalist ability of LLMs and VLMs from language-vision towards the 3D world.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, positive): “Recently, to achieve model scalability, Transformers-based (e.g., PointTransformer [78], Swin3d [79], PT2 [80], 3D-VisTA [81], LEO [82], PQ3D [83]) and Mamba-based (e.g., PointMamba [84], Mamba3D [85]) architectures have emerged.”
3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, positive): “Taking a step forward, recent works (Huang et al., 2023b; Hong et al., 2024) develop embodied foundation models that could plan and act in the 3D environment.”
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (2025, neutral): “…this limitation, a growing body of work has begun exploring robotics foundation models in the form of vision-language-action (VLA) models (Team et al., 2024; O’Neill et al., 2024; Brohan et al., 2023; Kim et al., 2024; Black et al., 2024; Bjorck et al., 2025; Li et al., 2024; Huang et al., 2024).”
ToolRL: Reward is All Tool Learning Needs (2025, neutral): “…a wide range of applications, including scientific discovery (Roohani et al., 2024; Inoue et al., 2024), research automation (Baek et al., 2024; Wang et al., 2024), embodied task completion (Zhang et al., 2023; Huang et al., 2023), and everyday decision-making (Ye et al., 2023; Zhai et al., 2024).”
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, positive): “…vision-language models (VLMs) to directly predict low-level robotic control actions, producing “vision-language-action” models (VLAs) [4, 33, 23, 22, 7, 15, 8, 50, 55, 51, 3, 2], which have demonstrated effective generalization to out-of-distribution test conditions and unseen semantic concepts.”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, background): “A number of recent works have explored similar recipes to ours and directly fine-tuned large pretrained VLMs for predicting robot actions [1, 7, 17, 18, 72–74].”