An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning

By Yaoyao Liu, Bernt Schiele, Qianru Sun. 2019

136 citations recorded by the source corpus.

Few-shot learning aims to train efficient predictive models with a few examples. The lack of training data leads to poor models that perform high-variance or low-confidence predictions. In this paper, we propose to meta-learn the ensemble of epoch-wise empirical Bayes models (E3BM) to achieve robust predictions. "Epoch-wise" means that each training epoch has a Bayes model whose parameters are specifically learned and deployed. "Empirical" means that the hyperparameters, e.g., used for learning and ensembling the epoch-wise models, are generated by hyperprior learners conditional on task-specific data. We introduce four kinds of hyperprior learners by considering inductive vs. transductive, and epoch-dependent vs. epoch-independent, in the paradigm of meta-learning. We conduct extensive experiments for five-class few-shot tasks on three challenging benchmarks: miniImageNet, tieredImageNet, and FC100, and achieve top performance using the epoch-dependent transductive hyperprior learner, which captures the richest information. Our ablation study shows that both "epoch-wise ensemble" and "empirical" encourage high efficiency and robustness in the model performance.

Subfields: Probabilistic & Bayesian Methods, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

1. Xiansheng Hua
2. Chi Zhang
3. Guosheng Lin
4. Yaoyao Liu
5. Qianru Sun
6. Bernt Schiele
7. Hanwang Zhang

In Meta & Continual Learning

1. Bernt Schiele
2. Yaoyao Liu
3. Qianru Sun
4. Chi Zhang
5. Guosheng Lin
6. Zhenyu Weng
7. Zhiping Lin
8. Huiping Zhuang

Citations in the corpus

• RMM: Reinforced Memory Management for Class-Incremental Learning (2023, neutral): “We can find some similar conclusions in few-shot learning that meta-learned hyperparameters can be transferred among different few-shot tasks [9].”
• Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, methodology): “Current few-shot methods mainly include optimization-based methods [12, 23, 32, 39, 45, 46, 55] and metric-based methods [13, 19, 44, 49, 52, 58, 57, 53].”
• Few-Shot Incremental Learning with Continually Evolved Classifiers (2021, methodology): “Optimization-based methods [8,16,24,28,29,36,37,50] and metric-based methods [9, 12, 35, 42, 46, 47, 51, 52] are two main lines of efforts.”
• Adaptive Aggregation Networks for Class-Incremental Learning (2020, background): “Technically, the network parameters are updated at one level and the key hyperparameters are updated at another level [13,21,24,45,46,52].”
• Interventional Few-Shot Learning (2020, background): “Thanks to them, the classification accuracy has been drastically increased [27, 72, 68, 35].”
• Fine-Grained Few-Shot Classification with Feature Map Reconstruction Networks (2020, unknown): “55 E3BM♥♦[ [13] ResNet-25 64.”
• GKEAL: Gaussian Kernel Embedded Analytic Learning for Few-Shot Class Incremental Task (2023, methodology): “Existing FSL methods are mainly optimization-based [5, 10, 16] and metric-based [6, 28, 29].”
• Free Lunch for Few-shot Learning: Distribution Calibration (2021, background): “Park et al. (2020); Wang et al. (2019); Liu et al. (2020b) tried to augment feature representations by leveraging intra-class variance.”