EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought

By Yao Mu, Qinglong Zhang, Mengkang Hu, Wen Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, Ping Luo. Neural Information Processing Systems, 2023

430 citations recorded by the source corpus.

Embodied AI is a crucial frontier in robotics, capable of planning and executing action sequences for robots to accomplish long-horizon tasks in physical environments. In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities. To achieve this, we have made the following efforts: (i) We craft a large-scale embodied planning dataset, termed EgoCOT. The dataset consists of carefully selected videos from the Ego4D dataset, along with corresponding high-quality language instructions. Specifically, we generate a sequence of sub-goals with the"Chain of Thoughts"mode for effective embodied planning. (ii) We introduce an efficient training approach to EmbodiedGPT for high-quality plan generation, by adapting a 7B large language model (LLM) to the EgoCOT dataset via prefix tuning. (iii) We introduce a paradigm for extracting task-related features from LLM-generated planning queries to form a closed loop between high-level planning and low-level control. Extensive experiments show the effectiveness of EmbodiedGPT on embodied tasks, including embodied planning, embodied control, visual captioning, and visual question answering. Notably, EmbodiedGPT significantly enhances the success rate of the embodied control task by extracting more effective features. It has achieved a remarkable 1.6 times increase in success rate on the Franka Kitchen benchmark and a 1.3 times increase on the Meta-World benchmark, compared to the BLIP-2 baseline fine-tuned with the Ego4D dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/00cb69a9f280317d1c59ac5827551ee9b10642b8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

An Embodied Generalist Agent in 3D World (2023, neutral): “We consider both Habitat-web (Ramrakhya et al., 2022) (fully supervised) and ZSON (Majumdar et al., 2022) (zero-shot) as baselines.”
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (2023, methodology): “While QAT maintains performance by simulating quantization during training, its training cost makes it unsuitable for LLM. PTQ techniques like AdaRound (Nagel et al., 2020) and BRECQ (Li et al., 2021) use gradient optimization to determine optimal rounding, but tuning all weights is time-intensive f”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “EmbodiedGPT [156] introduces the embodied-former, which outputs task-relevant instance-level features.”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, positive): “It can generate textual plans for multi-stage execution [39, 40], point trajectories [57], label bounding boxes of objects and gripper positions as additional observations [39], generate future image trajectories for open-loop following [32, 42], and generate fine-grained reward guidance for…”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “Other works have incorporated pre-trained language models, often either as an instruction encoder [63, 64, 43, 41, 44, 1, 65] or for high-level planning [66, 19, 17, 67, 68, 69].”
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, positive): “…82] and the feature extraction abilities of modern vision encoders [30, 65, 69], MLLMs, especially visual MLLMs, exhibit unprecedented visual understanding capabilities [34, 40, 78, 86, 91, 102, 103], a promising direction toward developing world models [50] and embodied agents [17, 21, 37, 59].”
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, background): “Recently, interesting works have made remarkable progress by extending LLM to audio, e.g. , KOSMOS-1 (Huang et al., 2023), X-LLM (Chen et al., 2023), PandaGPT (Su et al., 2023) and control systems like PaLM-E (Driess et al., 2023) and EmbodiedGPT (Mu et al., 2023)”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, positive): “Following previous works that leverage pre-trained language embed-dings [8, 40, 45, 103, 107–112] and pre-trained vision-language models [9, 113–115] in robotic imitation learning, we study both forms of pre-training in our experiments, specifically following the recipes of RT-1 [8] and RT-2 [9].”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “This includes creating models that can do OCR [12, 220], visual grounding [12, 26, 143, 189, 207, 212, 219], image-text-retrieval [97], additional languages [112], embodied AI tasks [17, 135, 140, 152] or leverage other expert systems [52].”
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, neutral): “Specifically, the participants rated the generated plans from different LVLM models using a scoring system similar to [58].”
Foundation models in robotics: Applications, challenges, and the future (2023, positive): “Em-bodiedGPT [70] utilizes preﬁx adapters to augment the 7B language model’s capacity to generate high-quality planning, training it on the EgoCOT dataset to avoid overly divergent language model responses.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, positive): “This integration also facilitates real-time feedback, enabling the agent to self-adjust strategies, thereby enhancing the robustness and adaptability of task execution [2], [190], [198].”
A survey on multimodal large language models (2023, neutral): “In order to further inspect what roles LLMs exactly play in LLM-Aided Visual Reasoning systems, existing related works are divided into three types: • LLM as a Controller • LLM as a Decision Maker”
ExpeL: LLM Agents Are Experiential Learners (2023, background): “Similarly, the ExpeL agent also autonomously learns from experiences using extracted insights and self-generated in-context trajectories by altering the execution prompt.”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “Additionally, PaLM-E [35] and EmbodiedGPT [83] represent advanced efforts in adapting VLLMs for embodied applications, significantly expanding their potential applications.”