AdapterHub: A Framework for Adapting Transformers

By Jonas Pfeiffer, Andreas Ruckl'e, Clifton A. Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych. 2020

753 citations recorded by the source corpus.

The current modus operandi in NLP involves downloading and fine-tuning pre-trained models consisting of millions or billions of parameters. Storing and sharing such large trained models is expensive, slow, and time-consuming, which impedes progress towards more general and versatile NLP methods that learn from and for many tasks. Adapters—small learnt bottleneck layers inserted within each layer of a pre-trained model— ameliorate this issue by avoiding full fine-tuning of the entire model. However, sharing and integrating adapter layers is not straightforward. We propose AdapterHub, a framework that allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages. The framework, built on top of the popular HuggingFace Transformers library, enables extremely easy and quick adaptations of state-of-the-art pre-trained models (e.g., BERT, RoBERTa, XLM-R) across tasks and languages. Downloading, sharing, and training adapters is as seamless as possible using minimal changes to the training scripts and a specialized infrastructure. Our framework enables scalable and easy access to sharing of task-specific models, particularly in low-resource scenarios. AdapterHub includes all recent adapter architectures and can be found at AdapterHub.ml

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

AdapterDrop: On the Efficiency of Adapters in Transformers (2020, methodology): “We compare the training and inference speed of full model ﬁne-tuning against the adapter architectures of Houlsby et al. (2019) and Pfeiffer et al. (2021a) (depicted in Figure 1) using the AdapterHub.ml framework (Pfeiffer et al., 2020a).”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, methodology): “…with the aim of learning a shared representation that will enable the model to generalize better on each task (Caruana, 1997; Collobert and Weston, 2008; Nam et al., 2014; Liu et al., 2016, 2017; Zhang and Yang, 2017; Ruder, 2017; Ruder et al., 2019; Sanh et al., 2019; Pfeiffer et al., 2020a).”
Training Neural Networks with Fixed Sparse Masks (2021, methodology): “The integration of FISH Masks across tasks and sharing amongst practitioners could also be a useful line of inquiry, as recent frameworks such as AdapterHub [39] have enabled for Adapter modules [19].”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, background): “Speciﬁcally through the use of adapter layers [18, 20, 34], small modules inserted in between the elements of a transformer layer which are ﬁnetuned instead of the model weights as a form of parameter efﬁcient ﬁne-tuning.”
Visual Prompt Tuning (2022, methodology): “One adapter module generally consists of a linear down-projection, followed by a nonlinear activation function, and a linear up-projection, together with a residual connection [59,60].”
Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “Implementation Details We use GPT-2 (Rad-ford et al., 2019) in HugginceFace Transformers (Wolf et al., 2020) as our backbone and adapter implementation by AdapterHub (Pfeiffer et al., 2020b).”
Memory Efficient Continual Learning with Transformers (2022, methodology): “For the Adapter implementation, we use Adapter-Hub [40], but no Adapter implementation was available for Vision Transformers.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “They act as additional network layers with a small number of parameters (Rebuffi et al., 2017a; Houlsby et al., 2019; Bapna and Firat, 2019; Pfeiffer et al., 2020) that reconfigure the original network on-the-fly for a target task, while keeping the parameters of the original network untouched and…”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, methodology): “ml repository (Pfeiffer et al., 2020b).”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, background): “Such task-specific adapters eliminate negative task interference by encapsulating task-specific information (Pfeiffer et al., 2020).”
Lightweight Adapter Tuning for Multilingual Speech Translation (2021, methodology): “…the use of adapters in NLP (Pfeif-fer et al., 2020b, 2021) and a framework built on top of HuggingFace Transformers library (Wolf et al., 2020) was also released to facilitate the down-loading, sharing, and adapting state-of-the-art pre-trained models with adapter modules (Pfeiffer et al., 2020a).”
What to Pre-Train on? Efficient Intermediate Task Selection (2021, methodology): “With the availability of model repositories (Wolf et al., 2020; Pfeiffer et al., 2020a) such approaches can be implemented without requiring additional data during model upload (i.e. in contrast to TaskEmbs , where the training dataset information needs to be made available).”
Parameter-efficient fine-tuning of large-scale pre-trained language models (2023, methodology): “Adapter tuning 13 is among the earliest approaches to steer pre-trained models with a limited number of parameters.”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “The Transformer architecture is first introduced in [81] and has re-energized the natural language processing (NLP) field from then on [27, 10].”
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021, methodology): “This is typically achieved through the use of adapter layers (Houlsby et al., 2019; Pfeiffer et al., 2020b).”
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2021, methodology): “We set the learning rate of fine-tuning and adapter-tuning to 2e-5 and 1e-4 according to prior studies (Pfeiffer et al., 2020; He et al., 2021).”
A Primer on Pretrained Multilingual Language Models (2021, positive): “In the context of the above discussion on adapters, we would like to point the readers to AdapterHub.ml 4 which is a useful repository containing all recent adapter architectures (Pfeiffer et al., 2020a).”