LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

By Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, W. Zhang, Pan Lu, Conghui He, Xiangyu Yue and 2 more. arXiv.org, 2023

758 citations recorded by the source corpus.

How to efficiently transform large language models (LLMs) into instruction followers is recently a popular research direction, while training LLM for multi-modal reasoning remains less explored. Although the recent LLaMA-Adapter demonstrates the potential to handle visual inputs with LLMs, it still cannot generalize well to open-ended visual instructions and lags behind GPT-4. In this paper, we present LLaMA-Adapter V2, a parameter-efficient visual instruction model. Specifically, we first augment LLaMA-Adapter by unlocking more learnable parameters (e.g., norm, bias and scale), which distribute the instruction-following ability across the entire LLaMA model besides adapters. Secondly, we propose an early fusion strategy to feed visual tokens only into the early LLM layers, contributing to better visual knowledge incorporation. Thirdly, a joint training paradigm of image-text pairs and instruction-following data is introduced by optimizing disjoint groups of learnable parameters. This strategy effectively alleviates the interference between the two tasks of image-text alignment and instruction following and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset. During inference, we incorporate additional expert models (e.g. captioning/OCR systems) into LLaMA-Adapter to further enhance its image understanding capability without incurring training costs. Compared to the original LLaMA-Adapter, our LLaMA-Adapter V2 can perform open-ended multi-modal instructions by merely introducing 14M parameters over LLaMA. The newly designed framework also exhibits stronger language-only instruction-following capabilities and even excels in chat interactions. Our code and models are available at https://github.com/ZrrSkywalker/LLaMA-Adapter.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/570079bbdd8758dfe865097e05719313c9c1301a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, methodology): “Specifically, inspired by LLaMA-Adapter V2 [27], the linear layer has been modified by introducing two learnable parameters including a bias β and a shift α into linear layers, as shown in Fig.”
Large Language Models are not Fair Evaluators (2023, background): “…performance across various tasks, leading to their widespread use as both the annotators (Peng et al., 2023; Xu et al., 2023) and evaluators (Zheng et al., 2023; Peng et al., 2023; Sun et al., 2023; Zhou et al., 2023; Gao et al., 2023; Wang et al., 2023b; Dubois et al., 2023; Wang et al., 2023a).”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “…emergent capabilities in various multimodal tasks such as perception ( e.g ., existence, count, position, OCR) [1,33,34,43,69,85], commonsense reasoning [26, 28, 33, 34, 36, 43, 61, 85], and code reasoning [21,24,26,39,41,78], resulting in a potential path to Artificial General Intelligence (AGI).”
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, neutral): “The evaluated MLLMs include BLIP-2 [30], LLaVA [33], MiniGPT-4 [56], mPLUG-Owl [50], LLaMA-Adapter-v2 [20], Otter [29], Multimodal-GPT [22], InstructBLIP [18], VisualGLM-6B [12], PandaGPT [42], ImageBind-LLM [2], and LaVIN [36].”
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, background): “…LLMs, such as Llama [78], Mistra [37], InternLM [77], QWen [65], GLM [25], and Baichuan [7], have empowered the customization of LLMs. Building on these open-source foundations, the community has seen substantial progress in multimodal large language models (MLLMs) [6, 21, 29, 48, 49, 82, 95, 100].”
CogAgent: A Visual Language Model for GUI Agents (2023, neutral): “…our model attained a score of 85.9, demonstrating superior handling of Vicuna-13B 22.4 - Otter [17] MPT-7B 24.7 - MiniGPT4 [44] Vicuna-13B 24.4 70.4 InstructBLIP [9] Vicuna-13B 25.6 77.3 LLaVA [21] LLaMA2-7B 28.1 66.3 LLaMA-Adapter v2 [14] LLaMA-7B 31.4 - DreamLLM [11] Vicuna-7B 35.9 76.5 LLaVA-1.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “A following work LLaMA-Adapter V2 [52] demonstrates that the simple multimodal fusion in LLaMA-Adapter cannot generalize to more challenging open-ended multimodal reasoning tasks, where the visual cues tend to dominate the adaptation prompts than the language instruction data.”
Personalize Segment Anything Model with One Shot (2023, background): “LLaMAAdapter [14, 58] proposes a zero-init attention to progressive incorporate new knowledge into foundation models, stabilizing the early-stage training.”
A survey on multimodal large language models (2023, positive): “Formulated in this way, these works transform LLMs into multimodal chatbots [13, 21, 22, 33, 35] and multimodal universal task solvers [23, 24, 26] through multimodal instruction tuning.”
IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (2023, methodology): “Recently, adapters have been utilized to achieve vision-language understanding for large language models [31, 32, 33, 34, 35].”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “Different from existing task-specific MLLM models (Ye et al., 2023; Peng et al., 2023b; Chen et al., 2023b; Liu et al., 2023d; Gao et al., 2023b), we integrate a diverse set of visual instruction tasks to tune the pre-trained model, aiming to acquire a wide range of capabilities.”
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models (2023, background): “Results of various open-source LMM on reported in the MM-VET paper [19] LLaMA-Adapter v2-7B [5] 16.”
A Comprehensive Overview of Large Language Models (2023, positive): “• The model can be fine-tuned to learn to call different external information resources and tools.”
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, methodology): “To broaden their applicability across diverse contexts, Multi-modal Large Language Models (MLLMs) [66, 20, 52, 11, 61, 70] have recently become The curve y = f(x) and the line y = -3, as shown in the figure, intersect at the points (0, -3), ( 𝑎 , -3), and ( 𝑏 , -3).”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, background): “Recent works [8, 9, 10] also explore adapting the pre-trained LLMs to a new domain by efficient tuning strategies like LoRA [11].”
Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, background): “Recently, Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023; Anil et al., 2023; Gao et al., 2023; Qwen, 2023) have attracted wide attention due to their powerful capabilities in text generation and comprehension.”
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, methodology): “Recent works like OpenFlamingo [11, 12], LLaVA [13] and others [14, 15, 16, 17, 18], have implemented this interface with promising initial results.”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, methodology): “We evaluate the models on M ATH V ISTA under three setups: (a) Text-Only LLMs including ChatGPT (OpenAI, 2022), GPT-4 (OpenAI, 2023a), and Claude-2 (Anthropic, 2023) in zero-shot and two-shot settings with Chain-of-Thought (CoT) (Wei et al., 2022b) and Program-of-Thought (PoT) (Chen et al., 2022b), ”