On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

By Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, Luo Si. 2021

255 citations recorded by the source corpus.

Adapter-based tuning has recently arisen as an alternative to fine-tuning. It works by adding light-weight adapter modules to a pretrained language model (PrLM) and only updating the parameters of adapter modules when learning on a downstream task. As such, it adds only a few trainable parameters per new task, allowing a high degree of parameter sharing. Prior studies have shown that adapter-based tuning often achieves comparable results to fine-tuning. However, existing work only focuses on the parameter-efficient aspect of adapter-based tuning while lacking further investigation on its effectiveness. In this paper, we study the latter. We first show that adapter-based tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. We then empirically compare the two tuning methods on several downstream NLP tasks and settings. We demonstrate that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Yang Liu
4. Iryna Gurevych
5. Wen-tau Yih
6. Jiawei Han
7. Mohit Bansal
8. Jianfeng Gao

Citations in the corpus

• Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, neutral): “…and Liang, 2021), Bit-fit (Ben Zaken et al., 2021), Child-Tuning (Xu et al., 2021), P-Tuning v2 (Liu et al., 2021b), LoRA (Hu et al., 2021), UnifiedSKG (Xie et al., 2022) and Adapter-based models (Rebuffi et al., 2017; Houlsby et al., 2019; Lin et al., 2020; He et al., 2021; Pfeiffer et al., 2021).”
• Adaptable Adapters (2022, background): “In addition to their parameter-efficiency, He et al. (2021) show that training adapter-layers (a) outperforms fine-tuning the whole model on low-data and cross-lingual settings, and (b) is more robust to overfitting.”
• Parameter-efficient fine-tuning of large-scale pre-trained language models (2023, background): “Specifically, adapter-based fine-tuning could perform better than vanilla fine-tuning on few-shot and cross-lingual scenarios 33 and is more robust under adversarial attacking 34 .”
• Is GPT-3 a Good Data Annotator? (2022, unknown): “…(Devlin et al., 2019; Peters et al., 2018; Yang et al., 2019; Raffel et al., 2020; Liu et al., 2021b), multi-task learning (Glo-rot et al., 2011; Liu et al., 2017), semi-supervised learning (Miyato et al., 2016), and few-shot learning (Deng et al., 2019; He et al., 2021; Qin and Joty, 2022a).”
• ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation (2023, background): “The success of adapters in NLP has also led to widespread interest in applying adapter techniques to visual tasks.”
• UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2021, methodology): “Recent studies (He et al., 2021; Li and Liang, 2021; Karimi Mahabadi et al., 2021) find that some PELT methods could be more effective than fine-tuning when the training data is limited, possibly due to the reduced risk of overfitting.”