MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
By Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny. 2023
3,250 citations recorded by the source corpus.
The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Furu Wei
- Luke S. Zettlemoyer
- Caiming Xiong
- Yoshua Bengio
- Yejin Choi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Yu Qiao
- Jitendra Malik
- Bernt Schiele
In Large Language Models
In Multimodal Learning
Citations in the corpus
- The False Promise of Imitating Proprietary LLMs (2023, methodology): “For example, the training process of many recent vision-language model (Li et al., 2022; Liu et al., 2023; Ye et al., 2023; Zhu et al., 2023) includes ChatGPT or GPT-4 outputs at some stages.”
- Improved Baselines with Visual Instruction Tuning (2023, methodology): “Qformer [31]) are used to reduce the number of visual patches [3, 13, 60].”
- RSGPT: A Remote Sensing Vision Language Model and Benchmark (2023, positive): “Inspired by the great success of MiniGPT-4 [84], we develop a Remote Sensing Generative Pretrained Model ( RSGPT ) based on finetuning InstructBLIP [12] on our newly created RSICap dataset.”
- Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, methodology): “The integration of LLMs with computer vision systems has given rise to Large Vision-Language Models (LVLMs) [6, 8, 22, 27, 28, 33, 40, 41, 49, 50, 55, 63].”
- NExT-GPT: Any-to-Any Multimodal LLM (2023, methodology): “And a series of open-source LLMs, e.g., Flan-T5 [13], Vicuna [12], LLaMA [80] and Alpaca [79], have greatly spurred advancement and made contributions to the community [109, 100].”
- ShareGPT4V: Improving Large Multi-Modal Models with Better Captions (2023, methodology): “Following the evolution, modality unification via LLMs becomes the inevitable tendency, and visual-aligned multi-modal LLMs[3, 5, 10, 30, 31, 35, 57, 60–62] have witnessed ever-changing advances in recent days.”
- mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “• To reduce the number of tokens, Mini-GPT4 (Zhu et al., 2023), mPLUG-Owl (Ye et al., 2023b), and Qwen-VL (Bai et al., 2023) adopt a structure similar to Q-Former (Li et al., 2023a), compressing the token count to a fixed size through learnable queries and cross-attention with visual features.”
- Generative Multimodal Models are In-Context Learners (2023, background): “With the remarkable progress in open-sourced LLMs, [42, 43, 49, 94] show promising results by connecting vision encoders and LLMs with a small intermediate model.”
- LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, positive): “Vision Transformers (ViTs) [12] are the most widely used vision encoder for LMMs, in which the input image is converted into a sequence of representative tokens by the ViT, and then fed into an LLM for understanding [18, 29, 52, 55].”
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “Our extensive evaluation under-scores the need for further advancements in handling longer multimodal data and we hope Video-MME will inspire future research and development in improving the capabilities of MLLMs.”
- A Comprehensive Overview of Large Language Models (2023, neutral): “Following this method, LLMs can be easily and effectively extended as multimodal chatbots [208], [202], [210] and multimodal task solvers [211], [212], [213].”
- SVIT: Scaling up Visual Instruction Tuning (2023, methodology): “…differentiable multimodal models, e.g. Flamingo (Alayrac et al., 2022), BLIP-2 (Li et al., 2023c), Kosmos (Huang et al., 2023; Peng et al., 2023), MiniGPT-4 (Zhu et al., 2023a), LLaVA (Liu et al., 2023b), InstructBLIP (Dai et al., 2023) which input both vision and language tokens into LLM.”
- Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “Vision large language models (VLLMs) [3, 5, 19, 21, 28, 69, 87, 113, 147], which leverage LLMs, have also achieved significant breakthroughs, enabling sophisticated vision-language dialogues and interactions.”