Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models

By Cheolhyoung Lee, Kyunghyun Cho, Wanmo Kang. 2019

239 citations recorded by the source corpus.

In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as "mixout", motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the average accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

How many data points is a prompt worth? (2021, background): “…checkpoint, roberta-large (355M parameters) from RoBERTa (Liu et al., 2019), which we load from the transformers (Wolf et al., 2020) library.2 In line with previous observations (McCoy et al., 2019; Dodge et al., 2020; Lee et al., 2020), head-based fine-tuning performance varies considerably.”
Making Pre-trained Language Models Better Few-shot Learners (2021, methodology): “A number of recent studies have focused on better methods for fine-tuning language models (Howard and Ruder, 2018; Dodge et al., 2020; Lee et al., 2020; Zhang et al., 2021).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “See Leibert et al. [2011] for an example of how one might design production infrastructure along these lines.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “This idea was used in [118] among other works, and although it often tends to underperform dropout [72], it is simple to implement.”
Revisiting Few-sample BERT Fine-tuning (2020, methodology): “While previous studies [8, 23, 33] focus on the validation performance, we split held-out test sets for our study.”
On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (2020, methodology): “Fine-tuning on CoLA was observed to be particularly stable in previous work (Phang et al., 2018; Dodge et al., 2020; Lee et al., 2020).”
PTR: Prompt Tuning with Rules for Text Classification (2021, background): “Besides fine-tuning language models for specific tasks, recent studies have explored better optimization and regularization techniques to improve fine-tuning (Lee et al., 2019; Dodge et al., 2020).”
Entailment as Few-Shot Learner (2021, methodology): “…improve the few-shot learning performance, including: (i) optimization and regularization techniques during the fine-tuning (Howard & Ruder, 2018; Lee et al., 2019; Zhang et al., 2020), (ii) semi-supervised learning to augment training data (Xie et al., 2020), and (iii) supervised contrastive…”
CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, background): “Since ﬁne-tuning large pre-trained models on the supervised down-stream tasks is often challenging (Dodge et al., 2020; Mosbach et al., 2020; Lee et al., 2020), we leave how to further improve the performance in the supervised setting as future work.”
Pretrained Language Model in Continual Learning: A Comparative Study (2022, methodology): “A natural question would be whether the later PLMs could better mitigate CF in continual learning (Mosbach et al., 2021; Lee et al., 2020a).”
Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors (2022, unknown): “While continual learning methods can update models sequentially, when used for sequential editing, they suffer from overfitting [25] and quickly forget previous edits and pre-training data [16, 22].”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “It is believed that catastrophic forgetting and the small size of datasets are possible reasons for fine-tuning instabilities [2], [301], [302].”
The MultiBERTs: BERT Reproductions for Robustness Analysis (2021, methodology): “In addition to the studies of robustness cited above, several authors have introduced methods to reduce BERT’s variability during finetuning (Zhang et al., 2021; Mosbach et al., 2021; Dodge et al., 2020; Lee et al., 2020; Phang et al., 2018).”
On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, methodology): “Mixout (Lee et al., 2019) proposes to directly reset a portion of the ﬁne-tuned model’s parameters to the pre-trained parameters with respect to a ratio.”
Adaptive Adapters: An Efficient Way to Incorporate BERT Into Neural Machine Translation (2021, background): “the enormous parameter scale of recent pre-trained language models [5], fine-tuning the whole model is parameter inefficient while being unstable and fragile on small datasets [16].”