Meta-Transfer Learning for Few-Shot Learning

By Qianru Sun, Yaoyao Liu, Tat-seng Chua, Bernt Schiele. 2018

1,232 citations recorded by the source corpus.

Meta-learning has been proposed as a framework to address the challenging few-shot learning setting. The key idea is to leverage a large number of similar few-shot tasks in order to learn how to adapt a base-learner to a new task for which only a few labeled samples are available. As deep neural networks (DNNs) tend to overfit using a few samples only, meta-learning typically uses shallow neural networks (SNNs), thus limiting its effectiveness. In this paper we propose a novel few-shot learning method called meta-transfer learning (MTL) which learns to adapt a deep NN for few shot learning tasks. Specifically, "meta" refers to training multiple tasks, and "transfer" is achieved by learning scaling and shifting functions of DNN weights for each task. In addition, we introduce the hard task (HT) meta-batch scheme as an effective learning curriculum for MTL. We conduct experiments using (5-class, 1-shot) and (5-class, 5-shot) recognition tasks on two challenging few-shot learning benchmarks: miniImageNet and Fewshot-CIFAR100. Extensive comparisons to related works validate that our meta-transfer learning approach trained with the proposed HT meta-batch scheme achieves top performance. An ablation study also shows that both components contribute to fast convergence and high accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “We notice that previous works [38, 44, 34, 47] have also leveraged the standard cross-entropy pre-training on the meta-training set.”
Backdoor Attacks on Federated Meta-Learning (2020, methodology): “We use 64 classes for meta-training (split among 4 users) and 20 classes for meta-testing (as in [25]); for each meta-training class, we hold out 20 examples for validation.”
Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, methodology): “Current few-shot methods mainly include optimization-based methods [12, 23, 32, 39, 45, 46, 55] and metric-based methods [13, 19, 44, 49, 52, 58, 57, 53].”
Learngene: From Open-World to Your Learning Task (2021, background): “Although meta domain adaptation (Bal-aji, Sankaranarayanan, and Chellappa 2018; Li et al. 2018; Sun et al. 2019) uses a lightweight model, it cannot continually learn new claases in an open-world environment.”
Few-Shot Incremental Learning with Continually Evolved Classifiers (2021, methodology): “Optimization-based methods aims to design efficient learning paradigm that enables fast network adaptation given limited data [8, 16, 28, 29, 37].”
Few-Shot Lifelong Learning (2021, methodology): “Few-shot learning methods generally employ meta-learning and metric learning techniques (Vinyals et al. 2016; Snell, Swersky, and Zemel 2017; Sung et al. 2018; Finn, Abbeel, and Levine 2017; Sun et al. 2019).”
Adaptive Aggregation Networks for Class-Incremental Learning (2020, methodology): “By differentiating the numbers of learnable parameters, we can have 3 block types: 1) “all” for learning all the convolutional weights and biases; 2) “scaling” for learning neuronlevel scaling weights [43] on the top of a frozen base model θbase; and 3) “frozen” for using only θbase (always frozen).”
Few-Shot Class-Incremental Learning (2020, methodology): “To achieve FSL, research studies usually adopt the metric learning and meta-learning strategies [43, 38, 40, 6, 39].”
Incremental Few-Shot Object Detection (2020, neutral): “Note that there are many more recent FSL methods [6, 26, 7, 17, 47, 55, 50, 27, 39, 11, 16] that produce better performance on image classification tasks than MAML.”
Mnemonics Training: Multi-Class Incremental Learning Without Forgetting (2020, background): “fer operations [31, 22] to train the network, rather than using standard weight over-writing.”
Meta-Learning in Neural Networks: A Survey (2020, background): “set of parameters to meta-learn, for example by subspace [76], [98], by layer [81], [98], [99], or by separating out scale and shift [100].”
Learning to Self-Train for Semi-Supervised Few-Shot Classification (2019, positive): “Network architectures of Θ and Φss are based on ResNet-12 (see details of MTL [32]) which consist of 4 residual blocks and each block has 3 CONV layers with 3 × 3 kernels.”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “nchmarks: miniImageNet, tieredImageNet and Fewshot-CIFAR100(FC100)1. 1. Introduction Few-shot learning aims to learn new concepts from a handful of training examples, e.g. from 1 or 5 training images [7, 42]. This ability is well-handled by humans, while in contrast, it remains challenging for machi”
Interventional Few-Shot Learning (2020, methodology): “For meta-learning, we deployed 5 representative methods that cover a large spectrum of meta-learning based FSL: 1) model initialization: MAML [18], 2) weight generator: LEO [51], transductive learning: SIB [27], 4) metric learning: MatchingNet (MN) [62], and 5) feature transfer: MTL [54].”
Image Generation From Small Datasets via Batch Statistics Adaptation (2019, background): “Meta-transfer learning [30] performs few-shot learning by updating only the scale and shift parameters, and achieves better performance than when fine-tuning all kernel parameters.”
Self-Supervised Learning for Few-Shot Image Classification (2019, neutral): “…68.88 ± 0.92% ProtoNet + [5] ResNet12 56.50 ± 0.40% 74.2 ± 0.20% CAML [9] ResNet12 59.23 ± 0.99% 72.35 ± 0.71% TPN [32] ResNet12 59.46% 75.65% MTL [33] ResNet12 61.20 ± 1.8% 75.50 ± 0.8% DN4 [30] ResNet12 54.37 ± 0.36% 74.44 ± 0.29% TADAM [10] ResNet12 58.50% 76.70% Qiao-WRN [8] Wide-ResNet28…”
Are We Overfitting to Experimental Setups in Recognition (2020, background): “Learning from large, uniform datasets [39, 28] has been the primary focus of supervised learning, although recently few-shot learning has become a popular subfield [36, 14, 30, 42].”