AdapterDrop: On the Efficiency of Adapters in Transformers

By Andreas Ruckl'e, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych. 2020

324 citations recorded by the source corpus.

Transformer models are expensive to fine-tune, slow for inference, and have large storage requirements. Recent approaches tackle these shortcomings by training smaller models, dynamically reducing the model size, and by training light-weight adapters. In this paper, we propose AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions. We show that AdapterDrop can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances. We further prune adapters from AdapterFusion, which improves the inference efficiency while maintaining the task performances entirely.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, methodology): “[23], which drops the adapters from lower transformer layers for a better training efficiency to T5 with ADAPTER.”
Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “For sequences containing dissimilar tasks where the transferable knowledge is limited and parameter sharing might cause degradation, Adapter+CL and Adapter+Drop seem more robust compared to Adapter+LAMOL and LAMOL, since they avoid catastrophic forgetting by parameter isolation.”
Parameter-efficient fine-tuning of large-scale pre-trained language models (2023, background): “In addition, the computational cost could be further reduced dynamically by removing adapters from lower transformer layers 32 .”
Adaptable Adapters (2022, background): “Rücklé et al. (2021) use layer dropping for adapter architectures.”
On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, methodology): “The tunable parameters for Adapter and BitFit are ﬁxed and cannot be changed.”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “We classify additive fine-tuning into three groups: Adapter-based Fine-tuning [9], [14], [15], [16], [17], [18], [19], [20], [21], [22], [76], in which the adapter module is incorporated into the transformer, allowing for fine-tuning without modifying the pretrained parameters, Soft Prompt-based…”
Structured Pruning Adapters (2022, background): “Finally, several works [33, 38, 41] explored the use of multi-task adapters.”
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021, background): “2According to Pfeiffer et al. (2020a, 2021a) and Rücklé et al. (2021), such an architecture with a single adapter per Transformer layer is more parameter-efficient while performing on par with the architecture of Houlsby et al. (2019) with two adapters per Transformer layer (one after the…”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “AdapterDrop [154], SparseAdapter [69], SPLoRA [72], LoRAPruning [216], ProPETL [210] PEFT Quantization BI-Adapter [87], PEQA [94], QLoRA [37], LoftQ [108], LQ-LoRA [60], QA-LoRA [195], INT2.”
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021, methodology): “Most previous work has demonstrated that it achieves comparable performance to fine-tuning (Bapna and Firat, 2019; Pfeiffer et al., 2020b,a,c; Rücklé et al., 2020; Wang et al., 2020; Guo et al., 2020).”
VeRA: Vector-based Random Matrix Adaptation (2023, background): “Another base-line, termed AdapterDrop by R¨uckl´e et al. (2021), enhances efficiency by omitting certain adapter layers and is represented as Adapter D .”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “…efficient setup based on adapters (Rebuffi et al., 2017; Houlsby et al., 2019; Stickland and Murray, 2019; Pfeiffer et al., 2020a,b,c,d; Lauscher et al., 2020a; Rücklé et al., 2020a,b, inter alia): additional parameter sets that are fine-tuned while the original pretrained model is kept frozen.”
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, background): “We investigate the effect of the size of adapters’ capacity by varying the reduction factor (also known as compression rate (Rücklé et al., 2021)) in the adapter’s bottleneck layer.”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, unknown): “PEFT encompasses strategies like partial fine-tuning [15, 16, 17, 18, 19, 20, 21, 22], soft prompt fine-tuning [23, 24, 25, 26, 27, 28, 29], non-linear adapter fine-tuning [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], and low rank adapter based fine-tuning [40, 41, 11, 42].”
Large Language Models Can Be Strong Differentially Private Learners (2021, methodology): “Adapter-based methods insert small subnetworks inside pretrained Transformers (Houlsby et al., 2019; Rücklé et al., 2020; Pfeiffer et al., 2020).”
Differentially Private Fine-tuning of Language Models (2021, background): “Some of these include random subspace projection (exploiting intrinsic dimensionality (Li et al., 2018; Aghajanyan et al., 2020)), preﬁx and prompt tuning (Li & Liang, 2021; Lester et al., 2021), tuning only biases (Cai et al., 2020; Ben Zaken et al., 2021), and other architecture variants including”