What Happens To BERT Embeddings During Fine-tuning?

By Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, Ian Tenney. 2020

222 citations recorded by the source corpus.

While much recent work has examined how linguistic information is encoded in pre-trained sentence representations, comparatively little is understood about how these models change when adapted to solve downstream tasks. Using a suite of analysis techniques—supervised probing, unsupervised similarity analysis, and layer-based ablations—we investigate how fine-tuning affects the representations of the BERT model. We find that while fine-tuning necessarily makes some significant changes, there is no catastrophic forgetting of linguistic phenomena. We instead find that fine-tuning is a conservative process that primarily affects the top layers of BERT, albeit with noteworthy variation across tasks. In particular, dependency parsing reconfigures most of the model, whereas SQuAD and MNLI involve much shallower processing. Finally, we also find that fine-tuning has a weaker effect on representations of out-of-domain sentences, suggesting room for improvement in model generalization.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, background): “This is because the fine-tuned BERT for a task captures highly task-specific information [41], which is difficult to be used by other tasks.”
CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks (2021, methodology): “However, BERT NCL (fine-tuning) is much worse than BERT ONE and Adapter-BERT NCL (adapter-tuning) as BERT fine-tuning learns highly task specific knowledge (Merchant et al., 2020).”
Meta-Learning with Sparse Experience Replay for Lifelong Language Learning (2020, background): “This perhaps relates to the ﬁnding by Merchant et al. (2020) that ﬁne-tuning BERT primarily affects the top layers and does not lead to catastrophic forgetting of linguistic phenomena in the deeper layers.”
DirectProbe: Studying Representations without Classifiers (2021, background): “Past work (Peters et al., 2019; Arase and Tsujii, 2019; Merchant et al., 2020) has shown that ﬁne-tuning pre-trained models on a speciﬁc task improves performance, and ﬁne-tuning is now the de facto procedure for using contextualized embeddings.”
Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data (2020, background): “Recent analyses of such MLM have shed light on the linguistic knowledge that is captured in the hidden states and attention maps (Clark et al., 2019b; Tenney et al., 2019a; Merchant et al., 2020).”
Revisiting Few-sample BERT Fine-tuning (2020, methodology): “[31] Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney.”
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021, background): “…et al., 2021), polysemy (Soler and Apidianaki, 2021), context-sensitivity (Reif et al., 2019), social bias (May et al., 2019; Bommasani et al., 2020), changes to the embedding space during ﬁne-tuning (Merchant et al., 2020), and as an evaluation metric for text generation (Zhang et al., 2020).”
Predicting Inductive Biases of Pre-Trained Models (2021, methodology): “This could be explored using intermediate task training (Pruksachatkun et al., 2020) in order to inﬂuence the extractability of features prior to ﬁne-tuning for the target task; e.g., Merchant et al. (2020) suggests ﬁne-tuning on parsing might improve the extractability of syntactic features.”
Do Vision Transformers See Like Convolutional Neural Networks? (2021, unknown): “These techniques have been very successful in providing insights on properties of different vision architectures [29, 22, 18], representation structure in language models [48, 25, 47, 21], dynamics of training methods [33, 24] and domain specific model behavior [27, 35, 38].”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, methodology): “This analysis has been extended to many downstream tasks, including text summarization [33], machine/speech translation [81, 86, 92], and various NLP tasks [36, 61].”
VAuLT: Augmenting the Vision-and-Language Transformer with the Propagation of Deep Language Representations (2022, background): “We also note the result that early layers of transformers change signiﬁcantly less compared to later layers during ﬁne-tuning (Merchant et al. 2020), potentially implying that ﬁne-tuning of the language model in VAuLT may be unnecessary.”
The neural architecture of language: Integrative modeling converges on predictive processing (2020, background): “5), consistent with recent results of models in high-level visual cortex (Geiger et al., 2020) as well as findings on the power of random initializations in natural language processing (Merchant et al., 2020; Tenney et al., 2019; Zhang & Bowman, 2018).”
Open-Source Embedding Models: A Comprehensive Survey of Techniques, Benchmarks, and Applications (2026, unknown): “Furthermore, fine-tuning pre-trained models on task-specific data can significantly enhance performance for particular downstream applications [68].”