VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

By Wenlong Huang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Li Fei-Fei. Conference on Robot Learning, 2023

1,011 citations recorded by the source corpus.

Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions. We present a large-scale study of the proposed method in both simulated and real-robot environments, showcasing the ability to perform a large variety of everyday manipulation tasks specified in free-form natural language. Videos and code at https://voxposer.github.io

View Paper: https://www.semanticscholar.org/paper/1cd8373490efc2d74c2796f4b2aa27c7d4415ec9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, unknown): “…and vision foundation models to enhance robotic capabilities, utilizing them as pretrained visual representations that accelerate robotic policy learning [29, 32, 30, 19, 31], for object localization in robotics tasks [9, 45], and for high-level planning and reasoning [1, 17, 42, 16, 43, 18, 6].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “VoxPoser [125] employs LLM and VLM to create two 3D voxel maps that represent affordance and constraint.”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, unknown): “Recently, vision-language-action (VLA) models have garnered signiﬁcant attention for their ability to extend pre-trained vision-languagemodelstoroboticsusinganext-tokenprediction approach.”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “VLMs can be used in various scenarios, especially robotics [4, 5, 15, 23, 37].”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “Following previous works that leverage pre-trained language embed-dings [8, 40, 45, 103, 107–112] and pre-trained vision-language models [9, 113–115] in robotic imitation learning, we study both forms of pre-training in our experiments, specifically following the recipes of RT-1 [8] and RT-2 [9].”
Executable Code Actions Elicit Better LLM Agents (2024, background): “…LLMs to acquire capabilities such as tool invocation and memory management (Mialon et al., 2023; Schick et al., 2023) and venture into real-world tasks such as controlling robots (Ahn et al., 2022; Huang et al., 2023; Ma et al., 2023) and performing scientific experiments (Bran et al., 2023).”
Octo: An Open-Source Generalist Robot Policy (2024, background): “Multiple works leverage diverse non-robot data or pretrained vision-language foundation models to boost policy generalization to new scenes and tasks [86, 103, 96, 16, 38, 11, 84, 36, 4, 37, 7, 3, 46, 15, 23].”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “…Adaptive Agent (AdA) [30], Palo et al. [15] Language-ImageGoal-Conditioned ValueLearning e.g. R3M [31], SayCan [32], Inner Monologue [33], VoxPoser [34], Mahmoudieh et al. [35], VIP [36], LIV [37], LOREL [38] High-LevelTaskPlanning e.g. NL2TL [39], Chen et al. [40] LLM-BasedCodeGeneration e.g.…”
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024, methodology): “We compare to VoxPoser [106] as a baseline.”
Language to Rewards for Robotic Skill Synthesis (2023, methodology): “Below we provide more details on the design of Reward Translatorand Motion Controller.”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “In these years, 3D vision has gained significant attention and development, driven by the rising popularity of autonomous driving [11, 69, 71], navigation [72, 76, 98], 3D scene understanding [2, 43, 46, 74], and robotics [31, 67].”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, methodology): “Recurrent Neural Networks (RNNs) [148] and Long Short-Term Memory (LSTM) networks [55] were used for various NLP tasks.”