SpotTune: Transfer Learning Through Adaptive Fine-Tuning

By Yunhui Guo, Humphrey Shi, Abhishek Kumar, Kristen Grauman, Tajana Šimunić, Rogério Schmidt Feris. 2018

517 citations recorded by the source corpus.

Transfer learning, which allows a source task to affect the inductive bias of the target task, is widely used in computer vision. The typical way of conducting transfer learning with deep neural networks is to fine-tune a model pretrained on the source task using data from the target task. In this paper, we propose an adaptive fine-tuning approach, called SpotTune, which finds the optimal fine-tuning strategy per instance for the target data. In SpotTune, given an image from the target task, a policy network is used to make routing decisions on whether to pass the image through the fine-tuned layers or the pre-trained layers. We conduct extensive experiments to demonstrate the effectiveness of the proposed approach. Our method outperforms the traditional fine-tuning approach on 12 out of 14 standard datasets. We also compare SpotTune with other state-of-the-art fine-tuning strategies, showing superior performance. On the Visual Decathlon datasets, our method achieves the highest score across the board without bells and whistles.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “, 2018), choosing layers to tune on a per-example basis (Guo et al., 2019), reinitializing layers over the course of training (Li et al.”
Dynamic Neural Networks: A Survey (2021, methodology): “For another example, the idea of dynamic routing is implemented for: 1) reducing the training cost under a multi-task setting [269] and 2) finding the optimal fine-tuning strategy for per example in transfer learning [270].”
Routing Networks with Co-training for Continual Learning (2020, background): “Sparsely-Gated Mixture-of-Experts (Shazeer et al., 2017) and other routing network variants (Rosenbaum et al., 2017; Veit & Belongie, 2018; Maziarz et al., 2019; Guo et al., 2019) can be trained by gradient descent.”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, background): “A general approach is to automatically learn which layers/parameters from a large-pretrained model to ﬁnetune and freeze (Guo et al., 2019), which we found is an exciting direction for future work.”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, background): “SpotTune [19] learns to adaptively route information through fine-tuned or pre-trained layers for different tasks.”
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “Unlike some algorithms such as BN, our SSF is not limited to the modulation of normalization layer, and it has a different motivation that is to alleviate the distribution mismatch between upstream tasks and downstream tasks for parameter-efficient fine-tuning.”
Learning Graph Structures With Transformer for Multivariate Time-Series Anomaly Detection in IoT (2021, methodology): “The core of the learning policy is named Gumbel-Softmax Sampling strategy [42], [43] inspired by the policy learning network in many reinforcement learning methods [44], [45].”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, background): “SpotTune (Guo et al., 2019) learns to adaptively route information through ﬁnetuned or pre-trained layers.”
A Broader Study of Cross-Domain Few-Shot Learning (2019, background): “In [18], the authors proposed an adaptive fine-tuning scheme to decide which layers of the pre-”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Dynamic computation to improve efﬁciency has been studied from multiple perspectives [1, 2, 50, 54, 15, 37, 13, 33].”
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition (2020, unknown): “We set τ = 5 as the initial value and gradually anneal it down to 0 during the training, as in [24].”
VA-RED2: Video Adaptive Redundancy Reduction (2021, background): “SpotTune (Guo et al., 2019) learns to adaptively route information through finetuned or pre-trained layers.”
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, methodology): “In transfer learning, SpotTune [17] learns to adaptively route information through finetuned or pre-trained layers.”
Gumbel-Matrix Routing for Flexible Multi-task Learning (2019, methodology): “Therefore, on the forward pass, we use the argmax to obtain a binary connection value, while on the backwards pass, we approximate it with softmax, similarly to (Guo et al., 2018).”
TranSlider: Transfer Ensemble Learning from Exploitation to Exploration (2020, methodology): “We compare our method with seven state-of-the-art knowledge transfer methods, including 1) fine-tuning [39], 2) l2-SP [21], 3) FitNet [32], 4) ABdistill [13], 5) CrossStitch [26], 6) Delta [22], 7) SpotTune [11], and three ensemble learning meth-ods with necessary modification for transfer learning ”
A New Benchmark for Evaluation of Cross-Domain Few-Shot Learning (2019, background): “In [11], the authors proposed an adaptive ﬁne-tuning scheme to decide which layers of the pretrained network should be ﬁne-tuned.”
Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation (2020, background): “ehaviors (i.e., no user profiles & no other item features), and whether such representations can be transferred to improve the downstream tasks, analogous to learned visual features from ImageNet [10, 20] or embedding features from text corpus [6, 15, 25]. Closely related to this work, [24] rece”
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (2022, unknown): “There are a variety of other fine-tuning heuristics (Ge & Yu, 2017; Guo et al., 2019; Zhang et al., 2020; Zhu et al., 2020; Jiang et al., 2021; Aghajanyan et al., 2021)—combining our insights with these ideas might lead to better methods.”