AdapterFusion: Non-Destructive Task Composition for Transfer Learning

By Jonas Pfeiffer, Aishwarya Kamath, Andreas Ruckl'e, Kyunghyun Cho, Iryna Gurevych. 2020

1,208 citations recorded by the source corpus.

Sequential fine-tuning and multi-task learning are methods aiming to incorporate knowledge from multiple tasks; however, they suffer from catastrophic forgetting and difficulties in dataset balancing. To address these shortcomings, we propose AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks. First, in the knowledge extraction stage we learn task specific parameters called adapters, that encapsulate the task-specific information. We then combine the adapters in a separate knowledge composition step. We show that by separating the two stages, i.e., knowledge extraction and knowledge composition, the classifier can effectively exploit the representations learned from multiple tasks in a non-destructive manner. We empirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that it effectively combines various types of knowledge at different layers of the model. We show that our approach outperforms traditional strategies such as full fine-tuning as well as multi-task learning. Our code and adapters are available at AdapterHub.ml.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, background): “For example, Zhang et al. (2020a) trains a “side” network that is fused with the pretrained model via summation; adapter-tuning inserts task-specific layers (adapters) between each layer of the pretrained LM (Houlsby et al., 2019; Lin et al., 2020; Rebuffi et al., 2017; Pfeiffer et al., 2020).”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “[31] propose a more efficient adapter variant, which keeps only one of the adapters in each layer for better training efficiency.”
WARP: Word-level Adversarial ReProgramming (2021, background): “Another approach, called Adapters (Houlsby et al., 2019; Pfeiffer et al., 2021), introduces new task-speciﬁc parameters that are added at every layer of the Transformer network.”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, methodology): “AdapterFusion (Pfeiffer et al., 2020a) leverages the knowledge of several adapters from different tasks and learns an optimal combination of the adapters’ output representations for a single target task (see Figure 3).”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, neutral): “…and Liang, 2021), Bit-ﬁt (Ben Zaken et al., 2021), Child-Tuning (Xu et al., 2021), P-Tuning v2 (Liu et al., 2021b), LoRA (Hu et al., 2021), UniﬁedSKG (Xie et al., 2022) and Adapter-based models (Rebufﬁ et al., 2017; Houlsby et al., 2019; Lin et al., 2020; He et al., 2021; Pfeiffer et al., 2021).”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “Various PEFT techniques have been explored, including prompt tuning [35, 30, 39, 38, 50, 72], Low-Rank Adaptation (LoRA) [23, 69, 20], and adapters [22, 48, 37, 9, 55].”
Visual Prompt Tuning (2022, background): “[63,64] exhaustively searched all possible configurations and found that only inserting adapters after the FFN “Add & LayerNorm” sub-layer works the best.”
Progressive Prompts: Continual Learning for Language Models (2023, background): “Of note, Qin & Joty (2021) report better performance of LFPT5 in continual learning setting than Adapter-Fusion (Pfeiffer et al., 2020) – a strong parameter-efﬁcient CL approach based on adapters.”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, background): “Prompts capture task-specific knowledge with much smaller additional parameters, than its competitors, such as Adapter [57, 42] and LoRA [16].”
Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “Implementation Details We use GPT-2 (Rad-ford et al., 2019) in HugginceFace Transformers (Wolf et al., 2020) as our backbone and adapter implementation by AdapterHub (Pfeiffer et al., 2020b).”
Memory Efficient Continual Learning with Transformers (2022, methodology): “AdapterFusion [39] provides state-of-the-art performance by composing the pre-trained Adapters and it can simply be repurposed for preventing CF in CL by learning one Adapter for every new task.”
Learning to Prompt for Continual Learning (2021, background): “Prompts capture task-specific knowledge with much smaller additional parameters, than its competitors, such as Adapter [43, 58] and LoRA [18].”
Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, positive): “AdapterFusion [42] used adapters proposed in [16].”
LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 (2021, methodology): “Moreover, LFPT5 is much more parameter-efﬁcient than AdapterFusion.”
Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (2021, methodology): “, 2019) and the pretrained language models adapting to multiple NLP tasks (Wang et al., 2020b; Pfeiffer et al., 2021).”
Continual Learning for Text Classification with Information Disentanglement Based Regularization (2021, background): “Related Learning Paradigms There exists some other learning paradigms also dealing with multiple tasks, such as multi-task learning (Yu et al., 2020) and transfer learning (Houlsby et al., 2019; Pfeiffer et al., 2021).”
AdapterHub: A Framework for Adapting Transformers (2020, methodology): “…from many more sources than was previously possible and makes research such as intermediate task training (Pruksachatkun et al., 2020), composing information from many tasks (Pfeiffer et al., 2020a), and training models for very low-resource languages (Pfeiffer et al., 2020b) much more accessible.”
Towards a Unified View of Parameter-Efficient Transfer Learning (2021, background): “Pfeiffer et al. (2021) have proposed a more efﬁcient adapter variant that is inserted only after the FFN “add & layer norm” sub-layer.”