The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)

By Zhengyuan Yang, Linjie Li, Kevin Qinghong Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, Lijuan Wang. 2023

973 citations recorded by the source corpus.

Large multimodal models (LMMs) extend large language models (LLMs) with multi-sensory skills, such as visual understanding, to achieve stronger generic intelligence. In this paper, we analyze the latest model, GPT-4V(ision), to deepen the understanding of LMMs. The analysis focuses on the intriguing tasks that GPT-4V can perform, containing test samples to probe the quality and genericity of GPT-4V's capabilities, its supported inputs and working modes, and the effective ways to prompt the model. In our approach to exploring GPT-4V, we curate and organize a collection of carefully designed qualitative samples spanning a variety of domains and tasks. Observations from these samples demonstrate that GPT-4V's unprecedented ability in processing arbitrarily interleaved multimodal inputs and the genericity of its capabilities together make GPT-4V a powerful multimodal generalist system. Furthermore, GPT-4V's unique capability of understanding visual markers drawn on input images can give rise to new human-computer interaction methods such as visual referring prompting. We conclude the report with in-depth discussions on the emerging application scenarios and the future research directions for GPT-4V-based systems. We hope that this preliminary exploration will inspire future research on the next-generation multimodal task formulation, new ways to exploit and enhance LMMs to solve real-world problems, and gaining better understanding of multimodal foundation models. Finally, we acknowledge that the model under our study is solely the product of OpenAI's innovative work, and they should be fully credited for its development. Please see the GPT-4V contributions paper for the authorship and credit attribution: https://cdn.openai.com/contributions/gpt-4v.pdf

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, background): “GPT-4V results are obtained from [51]. outperforms 80B IDEFICS [21], a Flamingo-like LMM with billions of trainable parameters for cross-modal connection.”
LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, neutral): “ShareGPT4V [9] provides multi-turn question-answering data from GPT-4V [61] interactions.”
Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, methodology): “While LVLMs like GPT-4V(ision) [48] and LLaVA-1.”
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, background): “Meanwhile, large multi-modal models (LMMs) that can understand both text and images have been making a major stride towards more general AI [9, 16, 32, 40, 73].”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “This section synthe-sizes the key lessons learned from this paradigm, analyzes the principles of effective instruction design, compares the tradeoffs between dominant fine-tuning methodologies, and outlines major open issues in the field.”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, methodology): “LLM models have been pre-trained on a huge amount of real-world datasets, and some LLMs, such as flamingo [136] and GPT-4V [149], have proved versatile capabilities on various vision-related tasks, e.g., using text to generate images, describing given images, and object detection.”
A survey on multimodal large language models (2023, neutral): “A new tool-related instruction dataset is collected and used to finetune the model.”
VILA: On Pre-training for Visual Language Models (2023, neutral): “Samples are taken from [60].”
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, background): “GPT-4V (OpenAI, 2023) introduces visual processing abilities to the GPT-4 models, unlocking many compelling abilities (Yang et al., 2023c;a).”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…in educational disciplines (Seo et al., 2015; Wang et al., 2017), helping analysts with logical queries about statistical data (Wu et al., 2023; Yang et al., 2023), and assisting in theorem proving and scientific discovery in advanced research fields (Taylor et al., 2022; Dong et al., 2023).”
GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, methodology): “WebAgent (Gur et al., 2023) proposes an enhanced planning strategy by summarizing the HTML documents and decomposing the instruction into multiple sub-instructions.”
OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, neutral): “Each question was annotated by at least three individuals, ensuring the validity of the questions-answer pairs.”
Ferret: Refer and Ground Anything Anywhere at Any Granularity (2023, background): “In a followup technical report (Yang et al., 2023), GPT-4V’s grounding ability is briefly touched.”
BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, background): “However, modern Multimodal LLMs often struggles with spatial concepts such as “left” and “right” [85].”
AppAgent: Multimodal Agents as Smartphone Users (2023, positive): “Recent studies [44, 47] have shown that GPT-4V can understand various types of images, including simple user interfaces (UIs) in popular smartphone apps.”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “Recent advancements have seen the creation of vision large language models (VLLMs) [3, 21, 56, 59, 62, 66, 100, 121, 128, 130, 136, 138, 141, 142, 148], which aim to enhance language models with the capability to process and interpret visual information.”
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback (2023, background): “By connecting visual signals and Large Language Models (LLMs), MLLMs show unprecedented capabilities in multimodal understanding, reasoning, and interaction [28, 29, 44].”