LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

By Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, Yu Qiao. 2023

1,040 citations recorded by the source corpus.

We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model. Using 52K self-instruct demonstrations, LLaMA-Adapter only introduces 1.2M learnable parameters upon the frozen LLaMA 7B model, and costs less than one hour for fine-tuning on 8 A100 GPUs. Specifically, we adopt a set of learnable adaption prompts, and prepend them to the word tokens at higher transformer layers. Then, a zero-initialized attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge. With our efficient training, LLaMA-Adapter can generate high-quality responses, comparable to Alpaca with fully fine-tuned 7B parameters. Besides language commands, our approach can be simply extended to multi-modal instructions for learning image-conditioned LLaMA model, which achieves superior reasoning performance on ScienceQA and COCO Caption benchmarks. Furthermore, we also evaluate the zero-initialized attention mechanism for fine-tuning other pre-trained models (ViT, RoBERTa) on traditional vision and language tasks, demonstrating the superior generalization capacity of our approach. Code is released at https://github.com/OpenGVLab/LLaMA-Adapter.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Quoc V. Le
4. Percy Liang
5. Luke S. Zettlemoyer
6. Hannaneh Hajishirzi
7. Caiming Xiong
8. Dacheng Tao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Xiaogang Wang
3. Dacheng Tao
4. Yu Qiao
5. Ming-Hsuan Yang
6. Trevor Darrell
7. Alan L. Yuille
8. Jitendra Malik

In Large Language Models

1. Jianfeng Gao
2. Saining Xie
3. Bo Li
4. Yuanhan Zhang
5. Chunyuan Li
6. Haotian Liu
7. Peng Jin
8. Pengcheng He

In Efficiency & Compression

1. Song Han
2. Ji Lin
3. Zhuang Liu
4. Haotian Tang
5. Jiaming Tang
6. Shang Yang
7. Xingyu Dang
8. Aojun Zhou

Citations in the corpus

• Visual Instruction Tuning (2023, methodology): “BLIP-2 a man is sitting on the back of a yellow cab”
• Instruction Tuning with GPT-4 (2023, methodology): “Furthermore, OpenFlamingo (Awadalla et al., 2023) and LLaMA-Adapter (Zhang et al., 2023) connect LLaMA with image inputs, paving a way to build open-source multi-modal LLMs.”
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…to judge LMMs on skills that require external knowledge (Schwenk et al., 2022; Shah et al., 2019), common sense reasoning (Zellers et al., 2019; Yin et al., 2021), scientific-knowledge (Lu et al., 2022; Kembhavi et al., 2017, 2016), medical understanding (Zhang et al., 2023b; Lau et al., 2018).”
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, background): “At their core, most MLLMs [8, 31, 71] are built on pre-trained vision [43, 54] and language [59, 68, 69] models.”
• EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, background): “Recent works [8, 9, 10] also explore adapting the pre-trained LLMs to a new domain by efficient tuning strategies like LoRA [11].”
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023, neutral): “, 2023), multi-modal Alpaca (LLaMA-Adapter) (Zhang et al., 2023).”
• Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, background): “Remarkable progress has been observed in recent large language models (LLMs) for various natural language processing tasks, with prominent examples such as GPT-3 [4], PaLM [7], LLaMA [56], ChatGPT [35], and the more recently proposed GPT-4 [36].”
• DriveLM: Driving with Graph Visual Question Answering (2023, methodology): “DriveLM-Agent employs a trajectory tokenizer that can be applied to any general VLM [40, 45, 53, 83], coupled with a graph prompting scheme that models logical dependencies as context inputs for VLMs.”
• SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities (2023, methodology): “We employ mHuBERT 1 as the speech tokenizer to discretize speech data into discrete units and remove the repetitive units of adjacent frames to get reduced units.”