MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning

By Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, Mohamed Elhoseiny. arXiv.org, 2023

726 citations recorded by the source corpus.

Large language models have shown their remarkable capabilities as a general interface for various language-related applications. Motivated by this, we target to build a unified interface for completing many vision-language tasks including image description, visual question answering, and visual grounding, among others. The challenge is to use a single model for performing diverse vision-language tasks effectively with simple multi-modal instructions. Towards this objective, we introduce MiniGPT-v2, a model that can be treated as a unified interface for better handling various vision-language tasks. We propose using unique identifiers for different tasks when training the model. These identifiers enable our model to better distinguish each task instruction effortlessly and also improve the model learning efficiency for each task. After the three-stage training, the experimental results show that MiniGPT-v2 achieves strong performance on many visual question-answering and visual grounding benchmarks compared to other vision-language generalist models. Our model and codes are available at https://minigpt-v2.github.io/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1ddbd08ad8cf22a5c66c4242194c4286328533bf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

OCRBench: on the hidden mystery of OCR in large multimodal models (2023, neutral): “MiniGPT4V2 [26] uses unique identifiers for different tasks when training the model to better distinguish each task instruction effortlessly.”
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, neutral): “(cid:2) DVG: LLaVA [10], Lynx [33], VPGTrans [9], MiniGPT-2 [57], Intel-LLaVA [58], and InternVL2-40B [59] (cid:2) LCVG: BLIP2 [5], InstructBLIP [14], InstructBLIP-T5 [14], and BLIVA [60] (cid:2) VCLG: LLaMA-Adapter V2 [8] (LA-V2), Otter-Image [13], and Cheetor [61] Table I provides a detailed…”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “Video & Audio [25%].”
GeoChat:Grounded Large Vision-Language Model for Remote Sensing (2023, methodology): “Notably, the recent MiniGPT-4-v2[4] fails to follow the instructions provided for this specific task and returns unrelated classes that are not a part of the dataset.”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, neutral): “Numerous works have been proposed in this direction [12], [13], [14], [15].”
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (2023, methodology): “Unlike previous studies that focused on limited tasks and pre-defined templates created by human experts [39; 8; 10], LRV-Instruction covers 16 vision-language tasks with open-ended instructions and answers, as shown in Fig.”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, background): “Besides, recent MLLMs are also extended to region-level parsing (Chen et al., 2023b; Peng et al., 2023b), in-context learning (Li et al., 2023a;b), arbitrary image resolutions (Bavishi et al., 2023), text-to-image generation (Wen et al., 2023; Dong et al., 2023), and 3D question answering (Xu et…”
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, methodology): “To broaden their applicability across diverse contexts, Multi-modal Large Language Models (MLLMs) [66, 20, 52, 11, 61, 70] have recently become The curve y = f(x) and the line y = -3, as shown in the figure, intersect at the points (0, -3), ( 𝑎 , -3), and ( 𝑏 , -3).”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “KOSMOS-2 [21], Shikra [22], MiniGPT-2 [23], CogVLM [24], SPHINX [25], and LLaVA [26] have enabled MLLMs to achieve region-based image understanding.”
BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, methodology): “Starting from LLaVA [48,50], people use LLM synthesized instruction-following chat data (which are in VQA format) for instruction tuning and achieve much better results [7,16,24,71].”
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “Later studies (Chen et al., 2023b; Peng et al., 2023; Wang et al., 2023; Chen et al., 2023a) have expanded LMM applications to broader multi-modal tasks, enhancing spatial perception through visual grounding.”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, background): “The high-level view of our approach is shown in Figure 2.”
EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, methodology): “Following most NLP models [22], [76], [77], the common language tokenizer is employed to encode language instructions.”
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, neutral): “More recently, with the advancements of MLLMs, their visual grounding capabilities on natural images have attracted significant attention (Bai et al., 2023; Chen et al., 2023a;b; Peng et al., 2023; Wang et al., 2024b; 2023; Ma et al., 2024).”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, neutral): “Some recent studies [2, 8, 9] incorporate grounding data and tune the LLM to enhance localizing and discriminating abilities.”
A Survey on Hallucination in Large Vision-Language Models (2024, methodology): “However, the restricted quantity of these tokens prevents encoding all the information present in images during the alignment process [ Yin et al. , 2023a; Chen et al. , 2023c ] , and consequent information loss raises the risk of hallucinations.”
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, background): “A series of studies [2, 5, 9, 10, 20, 21, 42, 46, 49, 62, 64, 80, 86, 92, 97, 98, 100] have explored further improve the MLLM in different dimensions, such as instruction tuning [11, 49, 98], efficient fine-tuning [33], high-resolution image inputs [6, 82, 83], hallucination mitigation [34, 87,…”