Meta-Transfer Learning Through Hard Tasks
By Qianru Sun, Yaoyao Liu, Zhaozheng Chen, Tat-seng Chua, Bernt Schiele. 2019
128 citations recorded by the source corpus.
Meta-learning has been proposed as a framework to address the challenging few-shot learning setting. The key idea is to leverage a large number of similar few-shot tasks in order to learn how to adapt a base-learner to a new task for which only a few labeled samples are available. As deep neural networks (DNNs) tend to overfit using a few samples only, typical meta-learning models use shallow neural networks, thus limiting its effectiveness. In order to achieve top performance, some recent works tried to use the DNNs pre-trained on large-scale datasets but mostly in straight-forward manners, e.g., (1) taking their weights as a warm start of meta-training, and (2) freezing their convolutional layers as the feature extractor of base-learners. In this paper, we propose a novel approach called meta-transfer learning (MTL), which learns to transfer the weights of a deep NN for few-shot learning tasks. Specifically, meta refers to training multiple tasks, and transfer is achieved by learning scaling and shifting functions of DNN weights (and biases) for each task. To further boost the learning efficiency of MTL, we introduce the hard task (HT) meta-batch scheme as an effective learning curriculum of few-shot classification tasks. We conduct experiments for five-class few-shot classification tasks on three challenging benchmarks, miniImageNet, tieredImageNet, and Fewshot-CIFAR100 (FC100), in both supervised and semi-supervised settings. Extensive comparisons to related works validate that our MTL approach trained with the proposed HT meta-batch scheme achieves top performance. An ablation study also shows that both components contribute to fast convergence and high accuracy.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Representation & Self-Supervised Learning
In Meta & Continual Learning
Citations in the corpus
- Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, methodology): “Current few-shot methods mainly include optimization-based methods [12, 23, 32, 39, 45, 46, 55] and metric-based methods [13, 19, 44, 49, 52, 58, 57, 53].”
- Few-Shot Incremental Learning with Continually Evolved Classifiers (2021, methodology): “Optimization-based methods [8,16,24,28,29,36,37,50] and metric-based methods [9, 12, 35, 42, 46, 47, 51, 52] are two main lines of efforts.”
- Adaptive Aggregation Networks for Class-Incremental Learning (2020, methodology): “Please note that φ0 is initialized with ones, following [42,43]; η0 is initialized with θbase; and α0 is initialized with 0.”
- Mnemonics Training: Multi-Class Incremental Learning Without Forgetting (2020, background): “ instance, Training GANs [6] can be formulated as a BOP with two optimization problems: maximizing the reality score of generated images and minimizing the real-fake classification loss. Meta-learning [5,15,31,32,38,39] is another BOP in which a meta-learner is optimized subject to the optimality of ”
- An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “An important difference is that we learn how to combine an ensemble of epoch-wise base-learners and how to generate efficient hyperparameters for base-learners, while other methods such as MAML [13], MAML++ [1], LEO [61], MTL [69,70], and SIB [25] use a single base-learner.”