MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action

By Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Qinghong Lin, E. Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, Lijuan Wang. 2023

603 citations recorded by the source corpus.

We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action. In this paper, we define and explore a comprehensive list of advanced vision tasks that are intriguing to solve, but may exceed the capabilities of existing vision and vision-language models. To achieve such advanced visual intelligence, MM-REACT introduces a textual prompt design that can represent text descriptions, textualized spatial coordinates, and aligned file names for dense visual signals such as images and videos. MM-REACT's prompt design allows language models to accept, associate, and process multimodal information, thereby facilitating the synergetic combination of ChatGPT and various vision experts. Zero-shot experiments demonstrate MM-REACT's effectiveness in addressing the specified capabilities of interests and its wide application in different scenarios that require advanced visual understanding. Furthermore, we discuss and compare MM-REACT's system paradigm with an alternative approach that extends language models for multimodal scenarios through joint finetuning. Code, demo, video, and visualization are available at https://multimodal-react.github.io/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, LLM Agents & Tool Use, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Percy Liang
  4. Yoshua Bengio
  5. Quoc V. Le
  6. Luke S. Zettlemoyer
  7. Zhiyuan Liu
  8. Yejin Choi

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Xiaogang Wang
  5. Trevor Darrell
  6. Yu Qiao
  7. Alan L. Yuille
  8. Jitendra Malik

In Prompting & Reasoning

  1. Hongsheng Li
  2. Zhiyuan Liu
  3. Peng Gao
  4. Renrui Zhang
  5. Pan Lu
  6. Yu Qiao
  7. Fahad Shahbaz Khan
  8. Ziwei Liu

In LLM Agents & Tool Use

  1. Song-Chun Zhu
  2. Maosong Sun
  3. Jianfeng Gao
  4. Michel Galley
  5. Luke S. Zettlemoyer
  6. Jifeng Dai
  7. Cheng Yang
  8. Ying Nian Wu

In Multimodal Learning

  1. Hongsheng Li
  2. Yu Qiao
  3. Jianfeng Gao
  4. Peng Gao
  5. Chunyuan Li
  6. Pan Lu
  7. Ping Luo
  8. Renrui Zhang

Citations in the corpus