Transfer Visual Prompt Generator across LLMs

By Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Tat-seng Chua. 2023

114 citations recorded by the source corpus.

While developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm. However, further tuning the VPG part of the MLLM still suffers from indispensable computational costs, i.e., requiring thousands of GPU hours and millions of training data. One alternative solution is to transfer an existing VPG from any existing MLLMs for the target MLLM. In this work, we for the first time investigate the VPG transferability across LLMs, and explore a solution to reduce the cost of VPG transfer. We first study the VPG transfer across different LLM sizes (e.g., small-to-large), and across different LLM types, through which we diagnose the key factors to maximize the transfer efficiency. Based on our observation, we design a two-stage transfer framework named VPGTrans, which is simple yet highly effective. Through extensive experiments, we demonstrate that VPGTrans helps significantly speed up the transfer learning process without compromising performance. Remarkably, it helps achieve the VPG transfer from BLIP-2 OPT$_\text{2.7B}$ to BLIP-2 OPT$_\text{6.7B}$ with over 10 times speed-up and 10.7% training data compared with connecting a VPG to OPT$_\text{6.7B}$ from scratch. Further, a series of intriguing findings and potential rationales behind them are provided and discussed. Finally, we showcase the practical value of our VPGTrans approach, by customizing two novel MLLMs, including VL-LLaMA and VL-Vicuna, with recently released LLaMA and Vicuna LLMs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension (2023, methodology): “In terms of specific evaluation dimension, MiniGPT-4 [7] model and mPLUG-Owl [9] model performs better in visual reasoning, while VPGTrans [40] model excels in action recognition and procedure understanding.”
OtterHD: A High-Resolution Multi-modality Model (2023, background): “As delineated by Zhang et al. [60], the architectures of current LMMs can be segmented into three components: a vision encoder, a projector, and a large language model (LLM).”
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, unknown): “To obtain end-to-end vision-language models, instruction tuning LLM on image-text pairs is proposed to align visual features with LLM and accomplish multimodal tasks in a unified way, for example, Flamingo (Alayrac et al., 2022), MiniGPT-4 (Zhu et al., 2023), LLaVA (Liu et al., 2023a), LLaMa-Adapter”
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, unknown): “2”
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, background): “Especially when dealing with complex prompts with multiple images (e.g., multi-modal chain of thoughts (Zhang et al., 2023b)), VLMs often overlook visual content when facing extensive text.”
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, unknown): “Subsequent studies have proposed advancements in vision-text interactions [5], reﬁned training methodologies [8], [9], and instruction tuning strategies [8], [9], [10], [11], [12], [13], [14].”
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding (2023, background): “In order to further explore the potential of LLMs, many researchers attempt to endow LLMs with the capability of understanding multi-modal content (Huang et al., 2023a; Zhang et al., 2023b; Yin et al., 2023).”
NExT-GPT: Any-to-Any Multimodal LLM (2023, background): “And a series of open-source LLMs, e.g., Flan-T5 [13], Vicuna [12], LLaMA [80] and Alpaca [79], have greatly spurred advancement and made contributions to the community [109, 100].”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “Aiming to address the computational complexity of training visual components in multi-modal models, Zhang et al. [130] introduce VPGTrans .”
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, background): “Mini-GPT4 (Zhu et al., 2023), mPLUG-OWL (Ye et al., 2023), VPGTrans (Zhang et al., 2023a), and InstructBLIP (Dai et al., 2023) retain Q-Former, replace language model to a larger one, and then tuning on meticulously collected instruction data.”
Generative Multimodal Models are In-Context Learners (2023, background): “A school of successive efforts [69, 76, 84, 90, 91] further improves visual instruction tuning with better overall training pipelines [6, 40], grounding annotations [14, 15, 85, 92], and extra tasks [6].”