PaLM-E: An Embodied Multimodal Language Model
By Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu and 12 more. 2023
2,915 citations recorded by the source corpus.
Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
- Quoc V. Le
- Trevor Darrell
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Bernt Schiele
In Large Language Models
In Multimodal Learning
In Robotics & Control
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Li Fei-Fei
- Yuke Zhu
- Suraj Nair
- Russ Tedrake
- Pierre Sermanet
Citations in the corpus
- Visual Instruction Tuning (2023, background): “PaLM-E [13] is an LMM for embodied AI.”
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action (2023, methodology): “These representations can be either discrete text words [13, 30, 33, 36] or continuous features projected into the textual feature space [2, 10, 14, 18, 27].”
- Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “Another category of research focus on learning multimodality abilities based on LLMs [7, 55, 56].”
- SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “…not only for visual understanding, but also for enabling practical applications in fields like robotics [10, 11] and augmented reality [12], where precise spatial awareness is crucial for tasks such as navigation [13], manipulation [10], and interaction with real-world environments [14].”
- Language Models can Solve Computer Tasks (2023, methodology): “First, our experiments use LLMs on HTML code, but ideally methods based on multimodal foundation models [13, 41, 2, 35] will be able to take actions based on text, images, audio, and video as input [3, 15, 33, 55].”
- Tool Learning with Foundation Models (2023, positive): “Similarly, PaLM-E (Driess et al., 2023) incorporates continuous inputs from different modalities into a PLM.”
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, methodology): “In this work, we adapt two previously proposed VLMs to act as VLA models: PaLI-X [16] and PaLM-E [17].”
- VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, background): “While Large Language Models (LLMs) are commonly used as foundational models across various modalities including language [3, 8], code [38, 43], audio [51], speech [1], and robotics [20, 86], the diffusion model remains the predominant approach for video generation.”
- Vision-Language Foundation Models as Effective Robot Imitators (2023, methodology): “While there have been some previous studies that incorporated large language models (LLMs) and vision-language models (VLMs) into robot systems as high-level planners (Ahn et al., 2022; Driess et al., 2023), making use of them directly for low-level control still poses challenges.”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “Most current robotic systems adopt such a hierarchical framework [9]–[11] because the high-level task planner can leverage models with high capacity while the low-level control policy can focus on speed and precision, similar to hierarchical reinforcement learning.”
- InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (2023, methodology): “Inspired by the success of PaLM-E [15] and BLIP2 [31], there is a promising approach to extending language models for vision-language tasks by leveraging vision features as extra inputs of LLMs.”
- CogVLM: Visual Expert for Pretrained Language Models (2023, neutral): “For instance, adapting an 8B parameter language model for VLM pretraining can lead to an 87.3% reduction in natural language generation (NLG) performance [Driess et al., 2023].”
- Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “Restrictions apply. models,havebeenextensivelytrainedwithlargedatasetsinstud-iesliketheRTseries[2],[9],PaLM-E[192],andMatcha[193] toachievealignmentofvisualandtextualfeaturesinembodied scenarios.”
- OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning (2024, background): “Recent efforts [40, 41, 42, 43, 44, 45, 46, 47] have focused on collecting robotic data using various embodiments for different tasks.”
- TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, background): “…literature that studies scaling of neural architectures for decision-making typically assume access to large datasets of near-expert demonstrations for behavior cloning (Reed et al., 2022; Lee et al., 2022; Kumar et al., 2022; Schubert et al., 2023; Driess et al., 2023; Brohan et al., 2023).”
- Transfer Visual Prompt Generator across LLMs (2023, background): “The VLLLMs advance in inheriting the great potentials of the backbone LLMs, and thus are capable of achieving excellent zero/few-shot performances [2, 18] on downstream tasks or be equipped with visual planning ability [11].”
- Qwen Technical Report (2023, background): “, 2023a), embodied lifelong learning or multimodal agents (Wang et al., 2023a; Driess et al., 2023), and multiple agents interacting with each other and even building a micro-society (Chen et al.”
- Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, background): “Other works di-2 rectly train language and image-based policies [6, 7, 55], some using VLM initializations [15, 52], to produce action sequences.”
- GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, positive): “Wit-nessing the power of LLM, the field of multimodal models (Yang et al., 2023c; Huang et al., 2023; Girdhar et al., 2023; Driess et al., 2023) is developing a new technology direction to leverage LLM as the universal interface to build general-purpose models, where the feature space of a specific…”
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, methodology): “The thriving of Large Language Model (LLM) has paved a new road to the multimodal field, i.e., Multimodal Large Language Model (MLLM) [40, 13, 26, 30, 19].”