Revisiting Fine-tuning for Few-shot Learning

By Akihiro Nakamura, Tatsuya Harada. 2019

67 citations recorded by the source corpus.

Few-shot learning is the process of learning novel classes using only a few examples and it remains a challenging task in machine learning. Many sophisticated few-shot learning algorithms have been proposed based on the notion that networks can easily overfit to novel examples if they are simply fine-tuned using only a few examples. In this study, we show that in the commonly used low-resolution mini-ImageNet dataset, the fine-tuning method achieves higher accuracy than common few-shot learning algorithms in the 1-shot task and nearly the same accuracy as that of the state-of-the-art algorithm in the 5-shot task. We then evaluate our method with more practical tasks, namely the high-resolution single-domain and cross-domain tasks. With both tasks, we show that our method achieves higher accuracy than common few-shot learning algorithms. We further analyze the experimental results and show that: 1) the retraining process can be stabilized by employing a low learning rate, 2) using adaptive gradient optimizers during fine-tuning can increase test accuracy, and 3) test accuracy can be improved by updating the entire network when a large domain-shift exists between base and novel classes.

Subfields: Artificial Intelligence, Computational Mechanics, Ocean Engineering, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Neil Houlsby
2. Xiaohua Zhai
3. Lucas Beyer
4. Alexander Kolesnikov
5. Joan Puigcerver
6. Quoc V. Le
7. Sylvain Gelly
8. Mostafa Dehghani

In Computational Mechanics

1. Alexander Kolesnikov
2. Joan Puigcerver
3. Sylvain Gelly
4. Xiaohua Zhai
5. Neil Houlsby
6. Lucas Beyer

In Ocean Engineering

1. Alexander Kolesnikov
2. Joan Puigcerver
3. Sylvain Gelly
4. Xiaohua Zhai
5. Neil Houlsby
6. Lucas Beyer

In Representation & Self-Supervised Learning

1. Neil Houlsby
2. Xiaohua Zhai
3. Lucas Beyer
4. Alexander Kolesnikov
5. Sylvain Gelly
6. Joan Puigcerver
7. Jyoti Prakash Sahoo
8. Jessica Yung

Citations in the corpus

• Big Transfer (BiT): General Visual Representation Learning (2019, background): “However, recent work has shown that a simple linear classifier on top of pre-trained representations or fine-tuning can attain similar or better performance [7,37].”
• Large Scale Learning of General Visual Representations for Transfer (2019, background): “However, recent papers have shown that a simple linear classifier on top of pre-trained representations or fine-tuning can attain similar or better performance [5,32].”
• A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “[74] replaced conventional gradient descent with an adaptive gradient optimizer, which both improve the fine-tuning process in the accuracy of the model.”
• TransPrompt: Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification (2021, methodology): “When the training data is scarce, the fine-tuned PLM would easily be over-fitted to specific instances (Nakamura and Harada, 2019).”