Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models

By Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, Jiaya Jia. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

397 citations recorded by the source corpus.

In this work, we introduce Mini-Gemini, a simple and effective framework enhancing multi-modality Vision Language Models (VLMs). Despite the advancements in VLMs facilitating basic visual dialog and reasoning, a performance gap persists compared to advanced models like GPT-4 and Gemini. We propose a novel approach to narrow the gap by mining the potential of VLMs for better performance across various cross-modal tasks. It tackles the following questions: (1) How can high-resolution visual tokens improve image understanding without lengthening the token sequence? (2) How to improve reasoning and generation abilities of VLM with high-quality data? (3) How to close the gap between open-source VLMs and proprietary models on reasoning-driven generation? In particular, to enhance visual tokens, we propose to utilize an additional visual encoder for high-resolution refinement without increasing the visual token count. We further construct a high-quality dataset that promotes precise image comprehension and reasoning-based generation, expanding the operational scope of current VLMs. In general, Mini-Gemini further mines the potential of VLMs and empowers current frameworks with image understanding, reasoning, and generation simultaneously. The proposed model supports a series of dense and MoE Large Language Models (LLMs) from 2B to 34B, which achieve leading performance in several zero-shot benchmarks and even surpasses the developed private models. It is demonstrated to attain 80.6% accuracy on the MMB benchmark (+5.4 vs Gemini Pro) and 74.1% on TextVQA (+4.6 vs LLaVA-NeXT), achieving leading performance in several zero-shot benchmarks and even surpasses the developed private models. Furthermore, Mini-Gemini is proven to improve consistently with stronger LLM, visual encoder, and data in experiments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b38845e9adbeeeab37519a2fc30e899411b4a36a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “Currently, there are two common technical routes: one involves designing a dual-branch image encoder [32, 72, 73, 116], and the other involves dividing a high-resolution image into many low-resolution tiles [33, 46, 52, 53, 60, 64, 121, 122].”
LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “Specifically, most focus on pushing the performance limits in single-image scenarios [26, 83, 172, 73, 163, 35], only a few recent papers have begun to explore multi-image scenarios [70, 47].”
SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (2024, neutral): “Furthermore, we check the consistency of SparseVLM on ConBench (Zhang et al., 2024b).”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, neutral): “5 (Zhang et al., 2024b)”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “Our current Cambrian-1 model uses a moderate number of visual tokens and does not adopt the any-resolution strategy [25, 70, 74] to handle ultra high-resolution images or those with extreme aspect ratios, which require a larger number of visual tokens.”