Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

By Chenfei Wu, Sheng-Kai Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, Nan Duan. 2023

869 citations recorded by the source corpus.

ChatGPT is attracting a cross-field interest as it provides a language interface with remarkable conversational competency and reasoning capabilities across many domains. However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. To this end, We build a system called \textbf{Visual ChatGPT}, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps. 3) providing feedback and asking for corrected results. We design a series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models. Our system is publicly available at \url{https://github.com/microsoft/visual-chatgpt}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Visual Instruction Tuning (2023, methodology): “( ii ) A system that coordinates various models via LangChain [1] / LLMs [35], such as Visual ChatGPT [53], X-GPT [63], MM-REACT [55], VisProg [18], and ViperGPT [46].”
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action (2023, methodology): “MM-REACT is closely related to the recent concurrent work of Visual ChatGPT [32] and ViperGPT [25].”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “Visual ChatGPT [60] uses pretrained models as tools for a chat agent, but does not generate programs and focuses on visual generation.”
OpenAGI: When LLM Meets Domain Experts (2023, methodology): “HuggingGPT [32] integrates the Hugging Face hub with task-specific models around ChatGPT to tackle generalized AI tasks.”
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality (2023, methodology): “However, systematic collaboration approaches, including Visual ChatGPT [Wu et al., 2023], MM-REACT [Yang et al., 2023], and HuggingGPT [Shen et al., 2023], are designed to facilitate the coordination of various vision models or tools to express visual information with text descriptions.”
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, methodology): “One approach focuses on leveraging existing pretrained task-specific tools such as image captioners [15], object detectors [61] and text-to-image generators [62] by either creating multimodal prompt interfaces [42, 63] or by executing LLM-generated programs [64, 65, 66].”
GAIA: a benchmark for General AI Assistants (2023, background): “…(Liu et al., 2023b) and AssistGPT (Gao et al., 2023) extending LLMs with planning components, Socratic Models (Zeng et al., 2022) or Visual ChatGPT (Wu et al., 2023) extended with multimodal models, WebGPT Nakano et al. (2021) fine-tuned for web-search, or a collection of tools and APIs, such as…”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “There are also re-searches [57, 60] that utilize the reasoning ability of LLMs and invoke pre-defined tools for visual applications.”
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, methodology): “7 WebGPT [34] 10 3 7 7 3 7 7 3 7 7 7 3 program 7 MM-ReAct [52] >10 3 7 7 3 7 3 3 3 3 3 3 word match 3 Visual ChatGPT [51] >10 3 - - 7 7 3 7 7 7 7 3 natural lang.”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Pre-training allows the models to align different modalities and enhance the representation learning ability of the model encoder.”
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face (2023, methodology): “Visual ChatGPT [26] fuses visual foundation models, such as BLIP [29] and Control-Net [30], with LLMs. Visual Programming [31] and ViperGPT [25] apply LLMs to visual objects by employing programming languages, parsing visual queries into interpretable steps expressed as Python code.”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, background): “Other efforts also extend LLMs for match problem solving (Wang et al., 2023a; Zhou et al., 2023), visual model system (Wu et al., 2023; Yang et al., 2023), and open-world recognition (Zhang et al., 2023b; Zhu et al., 2022).”
On the Challenges and Perspectives of Foundation Models for Medical Image Analysis (2023, neutral): “The applications are further expanded beyond textual domains with the integration of vision models OpenAI [2023], Wu et al. [2023c], Alayrac et al. [2022], Li et al. [2023a], Driess et al. [2023], Wang et al. [2023e], Liu et al. [2023b].”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “These methods are exemplified by VisualChatGPT [60], HuggingGPT [52], MM-REACT [65], and ViperGPT [54].”
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception (2024, methodology): “LLM-based agents Li et al. [2023], Liu et al. [2023a,b,c], Shen et al. [2023], Wu et al. [2023], Yang et al. [2023a], Shen et al. [2024], Yang et al. [2023b], Hong et al. [2023], Yang et al. [2023c], utilizing a variety of tools, have demonstrated strong capabilities in task planning and reasoning.”
The rise and potential of large language model based agents: a survey (2025, neutral): “For example, image processing and generation can be accomplished by an agent that draws on a visual model [248].”
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, methodology): “In recent years, with the continual expansion of LLMs’ capabilities, there has been a growing trend [14, 18, 21, 27, 37, 42, 51, 58, 61, 67, 69, 79] to combine them with visual foundation models and leverage their combined strengths to tackle more complex visionlanguage tasks.”
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction (2023, methodology): “Since this study focused on teaching the off-the-self language models to use tools instead of prompt engineering, we adopted a methodology outlined in the Visual ChatGPT [5] to construct tool-related prompts.”