Compacter: Efficient Low-Rank Hypercomplex Adapter Layers

2021

663 citations recorded by the source corpus.

Adapting large-scale pretrained language models to downstream tasks via fine-tuning is the standard method for achieving state-of-the-art performance on NLP benchmarks. However, fine-tuning all weights of models with millions or billions of parameters is sample-inefficient, unstable in low-resource settings, and wasteful as it requires storing a separate copy of the model for each task. Recent work has developed parameter-efficient fine-tuning methods, but these approaches either still require a relatively large number of parameters or underperform standard fine-tuning. In this work, we propose Compacter, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work. Compacter accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers. Specifically, Compacter inserts task-specific weight matrices into a pretrained model's weights, which are computed efficiently as a sum of Kronecker products between shared"slow"weights and"fast"rank-one matrices defined per Compacter layer. By only training 0.047% of a pretrained model's parameters, Compacter performs on par with standard fine-tuning on GLUE and outperforms standard fine-tuning on SuperGLUE and low-resource settings. Our code is publicly available at~\url{https://github.com/rabeehk/compacter}.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression

Citations in the corpus

• QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “…(PEFT) methods have been proposed such as prompt tuning [49, 33, 34], tuning the embedding layer inputs [1], tuning hidden states (IA 3 ) [37], adding full layers [27], tuning biases [71], learning a mask over weights based on Fisher information [55], and a combination of approaches [23].”
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Specifically, we compare with 9 strong PEFT methods: BitFit [47] which updates only the bias parameters; Adapters [23] which introduce task-specific layers after the self-attention and position-wise feed-forward networks; Compacter and Compacter++ [28] which improve upon adapters by using low-rank m”
• Training Neural Networks with Fixed Sparse Masks (2021, methodology): “Various works [19, 33, 32] have shown that, when appropriately designed, updating only the parameters in the adapters and the output layer can approach the performance of fine-tuning all parameters.”
• VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “…with Hyperformer, the number of trainable parameters of Hyperformer needs to be smaller than that of adapters, namely, | θ In general, we have [33] to solve the issues with the matrix decomposition and parameter sharing, and eventually, Compacter has been shown to have a better trade-off…”
• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Adapters [22] insert lightweight adaption modules into each layer of the pre-trained transformer and have been extended across numerous domains [19, 18, 70, 71].”
• The Efficiency Misnomer (2021, methodology): “…of Parameters: Number of trainable parameters is also used as an indirect indicator of computational complexity as well as memory usage (during inference) (Kim et al., 2021; Arnab et al., 2021; Tan and Le, 2019; Liu et al., 2021; Guo et al., 2020; Mahabadi et al., 2021a;b; Houlsby et al., 2019).”
• Progressive Prompts: Continual Learning for Language Models (2023, background): “Parameter-efficient Learning Recent works on parameter-efficient learning have shown that by training a subset of parameters, we can achieve a full model performance (Houlsby et al., 2019; Li & Liang, 2021; Karimi Mahabadi et al., 2021; Lester et al., 2021).”
• Towards a Unified View of Parameter-Efficient Transfer Learning (2021, background): “…that claimed comparable results to full fine-tuning on the GLUE benchmark with an encoder-only model (Guo et al., 2021; Ben Zaken et al., 2021; Mahabadi et al., 2021), or on relatively simple generation benchmarks such as E2E (Novikova et al., 2017) with an encoder-decoder model (Li & Liang,…”
• SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “Other work explores updating only small parts of the model (Zaken et al., 2021) or task-specific modules, such as adapters (Houlsby et al., 2019; Karimi Mahabadi et al., 2021) or low-rank structures (Mahabadi et al., 2021; Hu et al., 2021), while keeping the rest of the model fixed.”
• On Transferability of Prompt Tuning for Natural Language Understanding (2021, background): “…fine-tuning, which is friendly for tuning large PLMs. PT can achieve comparable performance with fine-tuning when the used PLM is extremely large, but the obvious performance gap still exists when the PLM is not so large, and PT also takes more memories than fine-tuning (Mahabadi et al., 2021).”
• Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022, background): “Many parameter-efficient adaption methods exist (Rebuffi et al., 2017; Houlsby et al., 2019; Karimi Mahabadi et al., 2021; Zaken et al., 2021; Hu et al., 2022) and some have shown strong performance under domain shift (Lester et al., 2021; Li and Liang, 2021).”
• Large Language Models Can Be Strong Differentially Private Learners (2021, background): “Methods that optimize low-rank matrices include the work by Hu et al. (2021); Mahabadi et al. (2021).”
• ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation (2023, background): “The success of adapters in NLP has also led to widespread interest in applying adapter techniques to visual tasks.”
• LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, methodology): “Parameter-efficient transfer learning (PETL) [31, 23, 39, 40, 34, 51, 58, 17, 24, 18, 42, 60, 14, 52, 59] is a recent research direction for online or multi-task learning.”
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023, methodology): “Compacter (Henderson et al., 2021) utilizes the Kronecker product, low-rank matrices, and parameter sharing across layers to generate adapter weights.”
• SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, background): “Another approach to compression is low-rank approximation, where each weight matrix is replaced with the product of two matrices with a smaller inner dimension, usually followed by a fine-tuning step (Hu et al., 2021; Mahabadi et al., 2021; Noach & Goldberg, 2020; Tukan et al., 2020).”
• DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2022, methodology): “Among PET techniques, low-rank adapters have received much attention because, in contrast to prompt-tuning techniques, low-rank adapters do not add to the sequence length, get trained faster, and perform better (Karimi Mahabadi et al., 2021).”