RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr

By Xingjian Li, Haoyi Xiong, Haozhe An, Chengzhong Xu, Dejing Dou. 2020

46 citations recorded by the source corpus.

Fine-tuning the deep convolution neural network(CNN) using a pre-trained model helps transfer knowledge learned from larger datasets to the target task. While the accuracy could be largely improved even when the training dataset is small, the transfer learning outcome is usually constrained by the pre-trained model with close CNN weights (Liu et al., 2019), as the backpropagation here brings smaller updates to deeper CNN layers. In this work, we propose RIFLE - a simple yet effective strategy that deepens backpropagation in transfer learning settings, through periodically Re-Initializing the Fully-connected LayEr with random scratch during the fine-tuning procedure. RIFLE brings meaningful updates to the weights of deep CNN layers and improves low-level feature learning, while the effects of randomization can be easily converged throughout the overall learning procedure. The experiments show that the use of RIFLE significantly improves deep transfer learning accuracy on a wide range of datasets, out-performing known tricks for the similar purpose, such as Dropout, DropConnect, StochasticDepth, Disturb Label and Cyclic Learning Rate, under the same settings with 0.5% -2% higher testing accuracy. Empirical cases and ablation studies further indicate RIFLE brings meaningful updates to deep CNN layers with accuracy improved.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiashi Feng
2. Ali Farhadi
3. Ludwig Schmidt
4. Jian Liang
5. Yifan Zhang
6. Mitchell Wortsman
7. Gabriel Ilharco
8. Percy Liang

In Computer Vision and Pattern Recognition

1. Jiashi Feng
2. Luc Van Gool
3. Trevor Darrell
4. Ali Farhadi
5. Dacheng Tao
6. Ming-Hsuan Yang
7. Andrea Vedaldi
8. Xiaogang Wang

In Representation & Self-Supervised Learning

1. Yifan Zhang
2. Jian Liang
3. Jiashi Feng
4. Ali Farhadi
5. Dapeng Hu
6. Simon Kornblith
7. Ludwig Schmidt
8. Bryan Hooi

In Optimization & Training

1. Jian Liang
2. Jiashi Feng
3. Dapeng Hu
4. Ludwig Schmidt
5. Mitchell Wortsman
6. Ali Farhadi
7. S. Gadre
8. Bryan Hooi

Citations in the corpus

• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “, 2019), reinitializing layers over the course of training (Li et al., 2020), or using multiple pretrained models with data-dependent gating (Shu et al.”
• Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, methodology): “As there is no fine-tuning method devoted to contrastive self-supervised models, we compare Core-tuning with advanced fine-tuning methods for general models ( e.g., supervised pre-trained models): L2SP [36], M & M [66], DELTA [34], BSS [9], RIFLE [35], SCL [19] and Bi-tuning [78].”