MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
By Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Qinghong Lin, E. Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, Lijuan Wang. 2023
603 citations recorded by the source corpus.
We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action. In this paper, we define and explore a comprehensive list of advanced vision tasks that are intriguing to solve, but may exceed the capabilities of existing vision and vision-language models. To achieve such advanced visual intelligence, MM-REACT introduces a textual prompt design that can represent text descriptions, textualized spatial coordinates, and aligned file names for dense visual signals such as images and videos. MM-REACT's prompt design allows language models to accept, associate, and process multimodal information, thereby facilitating the synergetic combination of ChatGPT and various vision experts. Zero-shot experiments demonstrate MM-REACT's effectiveness in addressing the specified capabilities of interests and its wide application in different scenarios that require advanced visual understanding. Furthermore, we discuss and compare MM-REACT's system paradigm with an alternative approach that extends language models for multimodal scenarios through joint finetuning. Code, demo, video, and visualization are available at https://multimodal-react.github.io/
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, LLM Agents & Tool Use, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Luke S. Zettlemoyer
- Zhiyuan Liu
- Yejin Choi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Yu Qiao
- Alan L. Yuille
- Jitendra Malik
In Prompting & Reasoning
In LLM Agents & Tool Use
- Song-Chun Zhu
- Maosong Sun
- Jianfeng Gao
- Michel Galley
- Luke S. Zettlemoyer
- Jifeng Dai
- Cheng Yang
- Ying Nian Wu
In Multimodal Learning
Citations in the corpus
- Visual Instruction Tuning (2023, methodology): “( ii ) A system that coordinates various models via LangChain [1] / LLMs [35], such as Visual ChatGPT [53], X-GPT [63], MM-REACT [55], VisProg [18], and ViperGPT [46].”
- A Comprehensive Overview of Large Language Models (2023, neutral): “In addition to CoT problems, LLMs can also be prompted with multimodal descriptions and tools, effectively dividing complex tasks into sub-tasks [224], [225].”
- EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, methodology): “Examples include models like Visual ChatGPT [33], MM-REACT [34], and HuggingGPT [35].”
- Tool Learning with Foundation Models (2023, positive): “Later, MM-ReAct (Yang et al., 2023) generalizes ReAct to the multi-modal domain by including several vision experts.”
- A survey on multimodal large language models (2023, neutral): “Specifically, in the ICL setting, LLMs learn from a few examples along with an optional instruction and extrapolate to new questions, thereby solving complex and unseen tasks in a few-shot manner [14,75,76].”
- VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, background): “One approach focuses on leveraging existing pretrained task-specific tools such as image captioners [15], object detectors [61] and text-to-image generators [62] by either creating multimodal prompt interfaces [42, 63] or by executing LLM-generated programs [64, 65, 66].”
- ExpeL: LLM Agents Are Experiential Learners (2023, background): “…game-playing, and workflows has surged, with emphasis on their world knowledge in fewshot settings (Ha, Florence, and Song 2023; Mu et al. 2023; Bran et al. 2023; Boiko, MacKnight, and Gomes 2023; Yang et al. 2023b; Lin et al. 2023a; Nakano et al. 2021; Wang et al. 2023b; Liu et al. 2023b).”
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, methodology): “In the realm of visual tools, various approaches have been proposed to enhance the capabilities of large language models in handling visual tasks [52, 51, 45, 11, 44], augmented with Hugging Face models [44], Azure models [52], visual foundation models [51].”
- GLaMM: Pixel Grounding Large Multimodal Model (2023, background): “The first encompasses models effective in textual responses but lacking in region-specific capabilities [6, 8, 22, 29, 51, 52, 61].”
- LISA: Reasoning Segmentation via Large Language Model (2023, background): “Moreover, numerous works [32, 44, 49, 52, 54] utilize prompt engineering, connecting independent modules via API calls, but without the benefits of end-to-end training.”
- Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, methodology): “The integration of LLMs with computer vision systems has given rise to Large Vision-Language Models (LVLMs) [6, 8, 22, 27, 28, 33, 40, 41, 49, 50, 55, 63].”
- ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “In addition, several works have extended this line of learning paradigm to other modalities [73, 69] and other domains [20].”
- LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, methodology): “In recent years, with the continual expansion of LLMs’ capabilities, there has been a growing trend [14, 18, 21, 27, 37, 42, 51, 58, 61, 67, 69, 79] to combine them with visual foundation models and leverage their combined strengths to tackle more complex visionlanguage tasks.”
- Chain of Thought Empowers Transformers to Solve Inherently Serial Problems (2024, background): “The success of CoT has even been extended to multi-modal QA (Zhang et al., 2023; Yang et al., 2023) and code generation (Chen et al., 2022; Gao et al., 2022).”
- A survey on large language model based autonomous agents (2023, neutral): “In the following, we introduce each of them more in detail.”