Investigating Transferability in Pretrained Language Models

By Alex Tamkin, Trisha Singh, D. Giovanardi, Noah D. Goodman. 2020

49 citations recorded by the source corpus.

How does language model pretraining help transfer learning? We consider a simple ablation technique for determining the impact of each pretrained layer on transfer task performance. This method, partial reinitialization, involves replacing different layers of a pretrained model with random weights, then finetuning the entire model on the transfer task and observing the change in performance. This technique reveals that in BERT, layers with high probing performance on downstream GLUE tasks are neither necessary nor sufficient for high accuracy on those tasks. Furthermore, the benefit of using pretrained parameters for a layer varies dramatically with finetuning dataset size: parameters that provide tremendous performance improvement when data is plentiful may provide negligible benefits in data-scarce settings. These results reveal the complexity of the transfer learning process, highlighting the limitations of methods that operate on frozen models or single data samples.

Subfields: Artificial Intelligence, Large Language Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Representation & Self-Supervised Learning

Citations in the corpus

Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Relatedly, Tamkin et al. (2020) ﬁnd a discrepancy between features f l ( x ) obtaining high probing performance, P ERF ( g, f, D O , D P ) , and features identiﬁed as important when ﬁne-tuning f while per-6 forming the probing task f l ( x ) 7→ z .”
What is being transferred in transfer learning? (2020, background): “Transfer learning is also a key components in recent breakthroughs in natural language processing, as pre-trained representations from task-agnostic transformer language models turned out to work extremely well on various downstream tasks [8, 34, 43, 28, 3], therefore extensively analyzed [19, 41].”
Rethinking embedding coupling in pre-trained language models (2020, background): “Overall, larger output embeddings prevent the model’s last layers from over-specializing to the pre-training task (Zhang et al., 2020; Tamkin et al., 2020), which enables training of more general Transformer models.”
What’s in Your Head? Emergent Behaviour in Multi-Task Transformer Models (2021, background): “A key challenge with probes is determining whether the information exists in the representation or is learned during probing (Hewitt and Liang, 2019; Tamkin et al., 2020; Talmor et al., 2020).”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Relatedly, Tamkin et al. (2020) ﬁnd a discrepancy between which features f l ( x ) obtain high probing performance, P ERF ( g, f, D O , D P ) , and which features are identiﬁed as important when ﬁne-tuning f while performing the probing task f l ( x ) 7→ z .”