Explicit Inductive Bias for Transfer Learning with Convolutional Networks

By Xuhong Li, Yves Grandvalet, Franck Davoine. 2018

447 citations recorded by the source corpus.

In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We show the benefit of having an explicit inductive bias towards the initial model, and we eventually recommend a simple $L^2$ penalty with the pre-trained model being a reference as the baseline of penalty for transfer learning tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “Surprisingly, we do not use any of the following forms of regularization during downstream tuning: weight decay to zero, weight decay to initial parameters [31], or dropout.”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “Other work has attempted to improve transfer learning by regularizing models toward their initialization (Xuhong et al., 2018), choosing layers to tune on a per-example basis (Guo et al., 2019), reinitializing layers over the course of training (Li et al., 2020), or using multiple pretrained models…”
Learning to Continually Learn (2020, background): “Other approaches similarly seek to modulate learning via task-specific synaptic importance [45], via the top-down attention mechanism of contrastive excitation backpropagation [20, 46], by interleaving pseudo examples generated from the current weights to limit plasticity [25], by employing L2 regul”
Incremental Learning Through Deep Adaptation (2017, neutral): “In [20], Li et al.”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “• L(2)-SP [25]: This is a recently proposed state-of-theart regularization method for fine-tuning.”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, methodology): “• L2-SP [23]: In addition to the prior methods, we test L2-SP, which is known to be effective for the classi-ﬁers.”
Deep learning-based electroencephalography analysis: a systematic review (2019, methodology): “In other cases, pre-trained models were used for a different purpose, such as object recognition, and were fine-tuned on the specific EEG task with the aim of providing a better initialization or regularization effect [97].”
On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning (2021, background): “Table 5 shows the results when we keep the weights frozen (Freeze), discouraging weights from changing with 2 -sp regularizer (Li et al., 2018, details in appendix), or standard ﬁne-tuning (ﬁne-tune).”
Zoo-Tuning: Adaptive Transfer from a Zoo of Models (2021, methodology): “Some works utilize knowledge of a pretrained model by regularizing the parameters (Xuhong et al., 2018) or features (Li et al.”
Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer Learning (2019, methodology): “Specially, L(2)-SP [20] considers that weight parameters should be driven to pre-trained values instead of the origin and takes the advantage of all pre-trained weights to refrain networks from forgetting useful information.”
MetaFSCIL: A Meta-Learning Approach for Few-Shot Class Incremental Learning (2022, background): “To avoid catastrophic forgetting, a promising solution is to modulate the plasticity of the learned weights [18, 30].”
Pay Attention to Features, Transfer Learn Faster CNNs (2020, methodology): “Li et al. (2018) proposed L 2 -SP regularization to minimize the L 2 -distance between each ﬁne-tuned parameter and its initial pre-trained value, thus preserving knowledge learned in the pre-trained model.”
Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, methodology): “Most existing methods for ﬁne-tuning, like L2-SP [36] and DELTA [34], are devised for supervised pre-trained models and tend to enforce some regularizer to prevent the ﬁne-tuned models changing too much from the pre-trained ones.”
Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification From the Bottom Up (2019, unknown): “1 L-SP [28] 87.”
TranSlider: Transfer Ensemble Learning from Exploitation to Exploration (2020, methodology): “Explicitly reusing the source models to facilitate the learning on the target domain has been widely used in transfer learning, such as parameter regularization [21] and knowledge distillation [14].”
RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr (2020, methodology): “To the best of our knowledge, RIFLE is yet the ﬁrst algorithmic regulariza-RIFLE tion method to improve the deep transfer learning algorithms based on explicit regularization such as (Li et al., 2018).”
Decoupled Mixup for Data-efficient Learning (2022, unknown): “L2-SP [25] promotes the similarity of the final solution with pre-trained weights by a simple L2 penalty.”