What is being transferred in transfer learning?

By Behnam Neyshabur, Hanie Sedghi, Chiyuan Zhang. 2020

631 citations recorded by the source corpus.

One desired capability for machines is the ability to transfer their knowledge of one domain to another where data is (usually) scarce. Despite ample adaptation of transfer learning in various deep learning applications, we yet do not understand what enables a successful transfer and which part of the network is responsible for that. In this paper, we provide new tools and analyses to address these fundamental questions. Through a series of analyses on transferring to block-shuffled images, we separate the effect of feature reuse from learning low-level statistics of data and show that some benefit of transfer learning comes from the latter. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.

Subfields: Artificial Intelligence, General Health Professions, Representation & Self-Supervised Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Health Professions

In Representation & Self-Supervised Learning

In Statistical Learning Theory

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “However, recent work (Neyshabur et al., 2020) observes that fine-tuned models optimized independently from the same initialization lie in the same basin of the error landscape, inspiring our method.”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Empirically, both can be attributed to obtaining a more robust representation [28], [89], [93], [191], and converging to a wider loss basin [28], [87], [93], [192], [193], suggesting a potential link among the sensitivity of representations, parameters and task-speciﬁc errors.”
FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, background): “Another line of work takes inspiration from transfer learning [28, 37] to tackle EFCIL.”
Disentangling Transfer in Continual Reinforcement Learning (2022, background): “Additionally, we show that the concept of feature reuse which is often utilized to explain supervised transfer learning [28, 33] might not be directly applicable in RL.”
Transfer without Forgetting (2022, background): “To sum up, the closer to the input of the model, the more easily transfer occurs from the pretraining dataset [39] thus incurring limited forgetting; however, (especially in final layers) the pretrained parameters require relevant modifications to fit the current training data.”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, positive): “To explain this behavior we build upon two separate lines of recent works—Hao et al. (2019) and Neyshabur et al. (2020) show that in the context of transfer learning, pre-trained weights lead to a ﬂat basin in the loss landscape when ﬁne-tuning on a single task.”
Representational Continuity for Unsupervised Continual Learning (2021, methodology): “Further, we measure the (cid:96) 2 distance between model parameters (Neyshabur et al., 2020) and visually compare the feature representations learned by different SCL and UCL strategies.”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “Similarly, a series of works has started a systematic empirical analysis of this phenomenon in deep networks (Parisi et al., 2019; Mirzadeh et al., 2020; Neyshabur et al., 2020; Nguyen et al., 2019; Ruder & Plank, 2017).”
Robust fine-tuning of zero-shot models (2021, background): “However, as similarly observed by previous work where part of the optimization trajectory is shared [25,42,71], we find that the zero-shot and fine-tuned models are connected by a linear path in weight-space along which accuracy remains high (explored further in Section A.”
How Well Self-Supervised Pre-Training Performs with Streaming Data? (2021, background): “The four domains in the sequence are assumed to have large distribution shifts among each other [44] and are widely used by transfer learning research [38].”
Editing Models with Task Arithmetic (2022, neutral): “[73] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang.”
Big Self-Supervised Models Advance Medical Image Classification (2021, methodology): “Given the small size of the validation dataset and following [34, 37] suggestion, for the downstream task evaluations we randomly re-split the training set into 67,429 training images, 22,240 validation images, and 33,745 test images.”
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (2022, background): “28, 41, 34, 93, 61, 49, 60], and study when and why transfer learning can be effective [104, 69, 1, 51, 54].”
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021, result): “1This notion of basin is also consistent with the definition proposed by Neyshabur et al. (2020).”
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth (2020, methodology): “These techniques have been very successful in analyzing deep learning, from properties of neural network training (Gotmare et al., 2018; Neyshabur et al., 2020), objectives (Resnick et al., 2019; Thompson et al., 2019; Hermann & Lampinen, 2020), and dynamics (Maheswaranathan et al., 2019) to…”
Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap (2022, background): “Starting from pre-trained weights, however, we expect less volatility [64], [65].”