Rethinking the Hyperparameters for Fine-tuning

By Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, Stefano Soatto. 2020

148 citations recorded by the source corpus.

Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyper-parameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several common practices of setting hyper-parameters for fine-tuning. Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks. (1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is a relatively unexplored parameter. We find that picking the right value for momentum is critical for fine-tuning performance and connect it with previous theoretical findings. (2) Optimal hyper-parameters for fine-tuning in particular the effective learning rate are not only dataset dependent but also sensitive to the similarity between the source domain and target domain. This is in contrast to hyper-parameters for training from scratch. (3) Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets. Our findings challenge common practices of fine- tuning and encourages deep learning practitioners to rethink the hyper-parameters for fine-tuning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, methodology): “For CUB, ResNet50 is pretrained on iNaturalist2017 [49], containing plants and animals and therefore a suitable source domain [32], using the backbone of [59].”
Forward Compatible Training for Large-Scale Embedding Retrieval Systems (2021, background): “Current practice for transfer learning largely centers around various fine-tuning schemes [24], which makes an assumption that the architecture used in the old domain and new domain are the same.”
LQF: Linear Quadratic Fine-Tuning (2020, methodology): “As the base model for linearization we use a ResNet-50 with Leaky-ReLU pre-trained on ImageNet.”
An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization (2020, background): “Li et al. (2020) considered hyperparameter selection when finetuning from one domain to another.”