On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines

By Marius Mosbach, Maksym Andriushchenko, Dietrich Klakow. 2020

475 citations recorded by the source corpus.

Fine-tuning pre-trained transformer-based language models such as BERT has become a common practice dominating leaderboards across various NLP benchmarks. Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process: training the same model with multiple random seeds can result in a large variance of the task performance. Previous literature (Devlin et al., 2019; Lee et al., 2020; Dodge et al., 2020) identified two potential reasons for the observed instability: catastrophic forgetting and a small size of the fine-tuning datasets. In this paper, we show that both hypotheses fail to explain the fine-tuning instability. We analyze BERT, RoBERTa, and ALBERT, fine-tuned on three commonly used datasets from the GLUE benchmark and show that the observed instability is caused by optimization difficulties that lead to vanishing gradients. Additionally, we show that the remaining variance of the downstream task performance can be attributed to differences in generalization where fine-tuned models with the same training loss exhibit noticeably different test performance. Based on our analysis, we present a simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than previously proposed approaches. Code to reproduce our results is available online: this https URL .

Subfields: Aerospace Engineering, Nuclear and High Energy Physics, Statistical and Nonlinear Physics, Large Language Models, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Nuclear and High Energy Physics

In Statistical and Nonlinear Physics

In Large Language Models

In Optimization & Training

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “More-over, despite the performance of Transformer-based models varying across random initializations (Dodge et al., 2019; 2020; Mosbach et al., 2020), we find that PVI estimates are quite stable: the correlation across seeds is r > 0 .”
How many data points is a prompt worth? (2021, background): “We follow recommendations of Mosbach et al. (2020) and Zhang et al. (2020) to train at a low learning rate (10−5) for a large number of steps (always at least 250, possibly for over 100 epochs).”
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021, background): “This usually happens with small training datasets (Dodge et al., 2020; Mosbach et al., 2021).”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Careful design and hyperparameter choices have a significant impact on performance and stability in limited data settings (Mosbach et al., 2021; Zhang et al., 2019).”
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, background): “As Mosbach et al. (2020) show, ﬁne-tuning BERT LARGE and RoBERTa BASE is a unstable due to vanishing gradients.”
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021, background): “Mosbach et al. (2020) show that standard ﬁne-tuning of pre-trained models can be unstable, which may be aggravated in our case as we are training on a diverse set of tasks simultaneously.”
ImageNet-21K Pretraining for the Masses (2021, background): “Previous works have observed that MLP-based models can be harder and less stable to use in transfer learning since they don’t have inherent translation inductive bias [6, 39, 35].”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, background): “Poor RoBERTa-large performance (for training on QuALITY only) is likely due to unstable training given our small training set (Mosbach et al., 2021).”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, methodology): “Our271 finetuning hyperparameters follow the recommen-272 dations of Mosbach et al. (2021): we use a learning273 rate of 2× 10−5, a learning rate warmup over the274 2FEVER-NLI builds upon the dataset from the FEVER shared task (Thorne et al., 2018).”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, methodology): “Empirical analysis suggests that ﬁne-tuning for longer, reinitializing top few layers (Zhang et al., 2020), and using debiased Adam optimizer during ﬁne-tuning (Mosbach et al., 2020) can make the ﬁne-tuning procedure more stable.”
CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, background): “Since ﬁne-tuning large pre-trained models on the supervised down-stream tasks is often challenging (Dodge et al., 2020; Mosbach et al., 2020; Lee et al., 2020), we leave how to further improve the performance in the supervised setting as future work.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “[303] showed that fine-tuning instability is not caused by any of these two and further showed that fine-tuning instability can be attributed to a) optimization difficulties which lead to vanishing gradients and b) generalization issues.”
Prompt-free and Efficient Few-shot Learning with Language Models (2022, background): “Handcrafted patterns are not included and we tune all parameters of the model. and unstable on resource-limited datasets (Dodge et al., 2020; Zhang et al., 2020; Mosbach et al., 2021).”
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020, background): “Catastrophic forgetting in NLP has raised increased attention recently (Mou et al., 2016; Arora et al., 2019; Chronopoulou et al., 2019).”
History Compression via Language Models in Reinforcement Learning (2022, background): “Previous work (Zhang et al., 2021; Mosbach et al., 2021) highlighted problems with finetuning PLTs in low-data regimes and proposed longer training times as remedy.”
Revisiting Transformer-based Models for Long Document Classification (2022, background): “However, the recent literature in text classi-ﬁcation mostly focuses on short sequences, such as sentences or paragraphs (Sun et al., 2019; Ad-hikari et al., 2019; Mosbach et al., 2021), which are sometimes misleadingly named as documents.”