Parameter-Efficient Transfer Learning for NLP

By Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly. 2019

6,877 citations recorded by the source corpus.

Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Graham Neubig
  6. Quoc V. Le
  7. Yejin Choi
  8. Yoshua Bengio

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Cordelia Schmid

In Natural Language Processing

  1. Mohit Bansal
  2. Graham Neubig
  3. Luke S. Zettlemoyer
  4. Yejin Choi
  5. Jianfeng Gao
  6. Noah A. Smith
  7. Percy Liang
  8. Kyunghyun Cho

In Efficiency & Compression

  1. Luke S. Zettlemoyer
  2. Mohit Bansal
  3. Quoc V. Le
  4. Song Han
  5. Yi-Lin Sung
  6. Barret Zoph
  7. Kurt Keutzer
  8. Jaemin Cho

Citations in the corpus