Revisiting Few-sample BERT Fine-tuning

By Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, Yoav Artzi. 2020

505 citations recorded by the source corpus.

We study the problem of few-sample fine-tuning of BERT contextual representations, and identify three sub-optimal choices in current, broadly adopted practices. First, we observe that the omission of the gradient bias correction in the BERTAdam optimizer results in fine-tuning instability. We also find that parts of the BERT network provide a detrimental starting point for fine-tuning, and simply re-initializing these layers speeds up learning and improves performance. Finally, we study the effect of training time, and observe that commonly used recipes often do not allocate sufficient time for training. In light of these findings, we re-visit recently proposed methods to improve few-sample fine-tuning with BERT and re-evaluate their effectiveness. Generally, we observe a decrease in their relative impact when modifying the fine-tuning process based on our findings.

Subfields: Computer Vision and Pattern Recognition, Materials Chemistry, Software, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Materials Chemistry

In Software

In Large Language Models

In Natural Language Processing

Citations in the corpus

Learning to Retrieve Passages without Supervision (2021, methodology): “Specifically, we train using Adam (Kingma and Ba, 2015) with bias-corrected moment estimates (Zhang et al., 2021), and a learning rate of 10−5 with warmup and linear decay.”
Improving and Simplifying Pattern Exploiting Training (2021, methodology): “We use the same pre-trained model and hyperparameters as PET, except we increased the number of training batches to 1k and choose the best checkpoint on the dev set, since it has been shown that training longer can help even with few samples (Zhang et al., 2021).”
How many data points is a prompt worth? (2021, background): “We follow recommendations of Mosbach et al. (2020) and Zhang et al. (2020) to train at a low learning rate (10−5) for a large number of steps (always at least 250, possibly for over 100 epochs).”
Making Pre-trained Language Models Better Few-shot Learners (2021, methodology): “small datasets can suffer from instability (Dodge et al., 2020; Zhang et al., 2020) and results may change dramatically given a new split of data.”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Careful design and hyperpa-rameter choices have a signiﬁcant impact on performance and stability in limited data settings (Mos-bach et al., 2021; Zhang et al., 2019).”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, methodology): “We fine-tune all methods for 3 epochs on large datasets and 20 epochs for low-resource datasets of GLUE (MRPC, CoLA, STS-B, RTE, BoolQ, CB, COPA, WiC) to allow the models to converge [27].”
GPT Understands, Too (2021, background): “The few-shot performance is sensitive to lots of factors (e.g., the order of training examples, random seed, and prompt patterns), and thus suffers from high variance (Zhao et al., 2021a; Lu et al., 2021; Zhang et al., 2020).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “Large-scale pre-trained language models (e.g., BERT (Devlin et al., 2019)) have demonstrated great ability to learn new tasks efficiently via fine-tuning (Zhang et al., 2021).”
Progressive Prompts: Continual Learning for Language Models (2023, methodology): “The predominant technique for adapting the language model to a downstream task T is finetuning, when all parameters Θ of the model (initialized from some pre-trained weights) are updated during task adaptation (Devlin et al., 2018; Zhang et al., 2020).”
Better Fine-Tuning by Reducing Representational Collapse (2020, methodology): “, 2019); Standard++ (STD++), a variant of standard fine-tuning that incorporates recently proposed best practices for fine-tuning, specifically longer fine-tuning and using bias correction in Adam (Zhang et al., 2020); and our proposed methods R3F and R4F.”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, methodology): “2 The original test sets are not publicly available, and following Zhang et al. (2021), for datasets fewer than 10K samples (RTE, MRPC, STS-B, CoLA), we divide the original validation set in half, using one half for validation and the other for the test.”
Entailment as Few-Shot Learner (2021, methodology): “…learning performance, including: (i) optimization and regularization techniques during the fine-tuning (Howard & Ruder, 2018; Lee et al., 2019; Zhang et al., 2020), (ii) semi-supervised learning to augment training data (Xie et al., 2020), and (iii) supervised contrastive learning as…”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, methodology): “Additionally, it has been recently demonstrated in NLP that ﬁne-tuning using cross entropy loss tends to be unstable (Zhang et al., 2020; Dodge et al., 2020), especially when supervised data is limited, a scenario in which pre-training is particularly helpful.”
Rethinking embedding coupling in pre-trained language models (2020, background): “Overall, larger output embed-dings prevent the model’s last layers from over-specializing to the pre-training task (Zhang et al., 2020; Tamkin et al., 2020), which enables training of more general Transformer models.”
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020, background): “10 Similary, a concurrent work (Zhang et al., 2020) has found that longer fine-tuning improve few-sample performance.”
Few-Shot Question Answering by Pretraining Span Selection (2021, methodology): “For fine-tuning, we use the hyperparameters from the default configuration of the HuggingFace Transformers package (Wolf et al., 2020).5 Specifically, we train all models using Adam (Kingma and Ba, 2015) with bias-corrected moment estimates for few-shot learning (Zhang et al., 2021).”