What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning
By Jaejun Lee, Raphael Tang, Jimmy J. Lin. 2019
153 citations recorded by the source corpus.
Pretrained transformer-based language models have achieved state of the art across countless tasks in natural language processing. These models are highly expressive, comprising at least a hundred million parameters and a dozen layers. Recent evidence suggests that only a few of the final layers need to be fine-tuned for high quality on downstream tasks. Naturally, a subsequent research question is, "how many of the last layers do we need to fine-tune?" In this paper, we precisely answer this question. We examine two recent pretrained language models, BERT and RoBERTa, across standard tasks in textual entailment, semantic similarity, sentiment analysis, and linguistic acceptability. We vary the number of final layers that are fine-tuned, then study the resulting change in task-specific effectiveness. We show that only a fourth of the final layers need to be fine-tuned to achieve 90% of the original quality. Surprisingly, we also find that fine-tuning all layers does not always help.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Juan Pino
- Changhan Wang
- Michael Auli
- Alexis Conneau
- Alexei Baevski
- Luke S. Zettlemoyer
- Graham Neubig
- Xian Li
Citations in the corpus
- Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, background): “Empirical studies shows that finetuning the final layers of BERT account for most of the quality gains on downstream tasks (Kovaleva et al., 2019; Lee et al., 2019).”
- UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2021, methodology): “Specifically, one may either train a subset of the model parameters such as the prediction head (Lee et al., 2019) and bias terms (Ben Zaken et al.”