VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

By Yi-Lin Sung, Jaemin Cho, Mohit Bansal. 2021

499 citations recorded by the source corpus.

Recently, fine-tuning language models pre-trained on large text corpora have provided huge improvements on vision-and-language (V&L) tasks as well as on pure language tasks. However, fine-tuning the entire parameter set of pre-trained models becomes impractical since the model size is growing rapidly. Hence, in this paper, we introduce adapter-based parameter-efficient transfer learning techniques to V&L models such as VL-BART and VL-T5. We evaluate our methods in a unified multi-task setup on both image-text and video-text benchmarks. For the image-text tasks, we use four diverse V&L datasets: VQAv2, GQA, NLVR2, and MSCOCO image captioning. For video-text tasks, we use TVQA, How2QA, TVC, and YC2C. With careful training and thorough experiments, we benchmark three popular adapter-based methods (Adapter, Hyperformer, Compacter) against the standard full fine-tuning and the recently proposed prompt-tuning approach. We also enhance the efficiency and performance of adapters by sharing their weights to attain knowledge across tasks. Our results demonstrate that training the adapter with the weight-sharing technique (4.18% of total parameters for image-text tasks and 3.39% for video-text tasks) can match the performance of fine-tuning the entire model. Lastly, we present a comprehensive analysis including the combination of adapter and task-specific prompts and the impact of V&L pre-training on adapters. 11The code for our CVPR 2022 paper is available at: https://github.com/ylsung/VL_adapter.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Efficiency & Compression

Citations in the corpus

ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “It has inspired a few recent works for language-augmented visual models [75, 59, 70, 18].”
CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “Subsequent work such as Parallel Adapter [19] and VL-Adapter [58] further extend [24]’s ability to language and multimodal tasks.”
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, background): “For example, Sung et al. (2022) and Alayrac et al. (2022) insert into the PLMs new layers that are trained from scratch.”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, background): “Moreover, researchers have increasingly delved into various PEFT techniques for computer vision [116], [117], as well as language-image and image-audio multimodal learning [118], [119], building upon PEFT methods in NLP [9], [11], [58].”
ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation (2023, background): “VL-Adapter [53] improves the efficiency and performance of adapters by sharing low-dimensional layers weights to attain knowledge across tasks.”
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, methodology): “Many successful PEFT approaches [10, 17, 22, 28, 31, 59, 60, 71, 73] have been proposed for adopting popular pre-trained models such as BERT [11], GPT [4, 53, 54], ViT [13], CLIP [52], and Stable Diffusion [55] to various downstream tasks.”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, methodology): “This technique is often used with CLIP [65] for transferring pretrained backbone models to different tasks [23, 65, 84, 93].”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “VL-Adapter [65] achieves the performance comparable to full fine-tuning on challenging vision-language tasks.”
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, background): “A common strategy to achieve this goal is to only train a small number of (extra) model parameters while freezing a large part of LLM [Hu et al., 2022, Sung et al., 2022].”
DoRA: Weight-Decomposed Low-Rank Adaptation (2024, methodology): “The result of LoRA and FT for both image/video-text tasks is directly quoted from (Sung et al., 2022).”
Vision Transformer Adapter for Dense Predictions (2022, methodology): “With the advent of CLIP [60], many CLIP-based adapters [24,69,81] were presented to transfer pre-trained knowledge to zero-shot or few-shot downstream tasks.”
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, methodology): “While most of the progress in PETL is made in NLP domain, researchers have also applied this technique to the CV [47, 46, 27, 59, 62, 28, 60, 21] and VL [52, 61, 62, 28, 60] domains.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “An earlier work VL-Adapter [168] directly applies several PEFT methods (Adapter [75], Hyperformer [129] and Compacter [91]) on VL-BART [26] then benchmarks them on several image-text and video-text tasks.”