PaLM-E: An Embodied Multimodal Language Model

By Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu and 12 more. 2023

2,915 citations recorded by the source corpus.

Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Jianfeng Gao
  3. Mohit Bansal
  4. Yoshua Bengio
  5. Percy Liang
  6. Quoc V. Le
  7. Trevor Darrell
  8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Bernt Schiele

In Large Language Models

  1. Furu Wei
  2. Li Dong
  3. Saining Xie
  4. Yu Qiao
  5. Ping Luo
  6. Zhiyuan Liu
  7. Xipeng Qiu
  8. Maosong Sun

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Linjie Li
  4. Zicheng Liu
  5. Jianfeng Gao
  6. Jianfeng Wang
  7. Yu Qiao
  8. Hongsheng Li

In Robotics & Control

  1. Sergey Levine
  2. Chelsea Finn
  3. Pieter Abbeel
  4. Li Fei-Fei
  5. Yuke Zhu
  6. Suraj Nair
  7. Russ Tedrake
  8. Pierre Sermanet

Citations in the corpus