Better Fine-Tuning by Reducing Representational Collapse

By Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke S. Zettlemoyer, Sonal Gupta. 2020

241 citations recorded by the source corpus.

Although widely adopted, existing approaches for fine-tuning pre-trained language models have been shown to be unstable across hyper-parameter settings, motivating recent work on trust region methods. In this paper, we present a simplified and efficient method rooted in trust region theory that replaces previously used adversarial objectives with parametric noise (sampling from either a normal or uniform distribution), thereby discouraging representation change during fine-tuning when possible without hurting performance. We also introduce a new analysis to motivate the use of trust region methods more generally, by studying representational collapse; the degradation of generalizable representations from pre-trained models as they are fine-tuned for a specific end task. Extensive experiments show that our fine-tuning method matches or exceeds the performance of previous trust region methods on a range of understanding and generation tasks (including DailyMail/CNN, Gigaword, Reddit TIFU, and the GLUE benchmark), while also being much faster. We also show that it is less prone to representation collapse; the pre-trained models maintain more generalizable representations every time they are fine-tuned.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “For CM3 we look at three settings for fine-tuning; standard finetuning, better fine-tuning using adversarial methods (Aghajanyan et al., 2020), and better fine-tuning over prompts derived from Aghajanyan et al.”
HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, methodology): “Given the recent advancements in ﬁnetuning, we also report results us-ing the recently proposed R3F method for ﬁnetun-ing (Aghajanyan et al., 2020a) for both RoBERTa and HTLM .”
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020, background): “Moreover, standard approaches towards ﬁne-tuning seem to have non-trivial effects on the generalization of pre-trained representations (Aghajanyan et al., 2020).”
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021, methodology): “Therefore, we employ the R3F/R4F methods (Aghajanyan et al., 2020) to combat this issue.”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, background): “It is also difficult to adapt to new domains without forgetting the original data (Aghajanyan et al., 2021) or restrict access to certain domains the LM has been exposed to during training (e.”
Robust fine-tuning of zero-shot models (2021, methodology): “…processing, previous work proposed stable ﬁne-tuning methods that incur computational overhead (Jiang et al., 2019; Zhu et al., 2020), alleviating problems such as representational collapse (Aghajanyan et al., 2021 2021) explored the connection between quadratic regularization and interpolation.”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, methodology): “Similarly, Aghajanyan et al. (2020) propose a regularized ﬁne-tuning procedure inspired by trust-region theory that replaces adversarial objectives with parametric noise sampled from normal or uniform distribution in order to prevent representation collapse during ﬁne-tuning for better…”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, methodology): “The smaller GLUE tasks (CoLA, RTE, MPRC, and STS-B) are unstable: Many pretraining research omit them and more advanced finetuning strategies are required to achieve stable evaluations (Aghajanyan et al., 2020).”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, unknown): “A number of mechanisms have been proposed for increasing robustness against catastrophic forgetting [28, 29, 30, 31, 32, 33].”
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022, methodology): “We also show that this observation holds in a standard multi-task training regime, without the optimization tricks used Aghajanyan et al. (2021), namely loss scaling and regularized ﬁnetuning (Aghajanyan et al., 2020).”
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (2021, methodology): “Some of them utilize the pretrained weights to regularize the deviation of the fine-tuned model (Lee et al., 2020; Daumé III, 2007; Chen et al., 2020), while others compress the output information (Mahabadi et al., 2021) or injects noise into the input (Jiang et al., 2020; Aghajanyan et al., 2021).”
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (2022, unknown): “There are a variety of other fine-tuning heuristics (Ge & Yu, 2017; Guo et al., 2019; Zhang et al., 2020; Zhu et al., 2020; Jiang et al., 2021; Aghajanyan et al., 2021)—combining our insights with these ideas might lead to better methods.”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, background): “…from the target domain (Howard and Ruder, 2018; Gururangan et al., 2020), intermediate-task ﬁne-tuning (Phang et al., 2019), multi-task pre-ﬁnetuning (Aghajanyan et al., 2021a), better de-sign choices and training strategies (Mosbach et al., 2021; Zhang et al., 2021), and regularization-oriented…”
Taming Sparsely Activated Transformer with Stochastic Experts (2021, background): “• R3F (Aghajanyan et al., 2020) uses a regularizer to reduce representational collapse.”
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021, background): “Aghajanyan et al. (2021) regularize fine-tuning by introducing noise to the input which is similar to adversarial training for fine-tuning studied in Zhu et al. (2020).”
Improving language models fine-tuning with representation consistency targets (2022, methodology): “Local smoothness inducing regularization R3F (Aghajanyan et al., 2020a) prevents aggressive model updates by restricting divergence of model outputs upon input perturbation.”
R-Drop: Regularized Dropout for Neural Networks (2021, methodology): “Speciﬁcally, our result also surpasses the PEGASUS method [74], which brings a novel self-supervised paradigm carefully designed for summarization, and the previous best work BART+R3F [1], which introduces a parametric noise sampled from normal or uniform distributions.”