Diversity With Cooperation: Ensemble Methods for Few-Shot Classification

By Nikita Dvornik, Cordelia Schmid, Julien Mairal. 2019

210 citations recorded by the source corpus.

Few-shot classification consists of learning a predictive model that is able to effectively adapt to a new class, given only a few annotated samples. To solve this challenging problem, meta-learning has become a popular paradigm that advocates the ability to ``learn to adapt''. Recent works have shown, however, that simple learning strategies without meta-learning could be competitive. In this paper, we go a step further and show that by addressing the fundamental high-variance issue of few-shot learning classifiers, it is possible to significantly outperform current meta-learning techniques. Our approach consists of designing an ensemble of deep networks to leverage the variance of the classifiers, and introducing new strategies to encourage the networks to cooperate, while encouraging prediction diversity. Evaluation is conducted on the mini-ImageNet, tiered-ImageNet and CUB datasets, where we show that even a single network obtained by distillation yields state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Meta & Continual Learning

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, neutral): “53 Diversity w/ Cooperation [11] ResNet-18 59.”
Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, methodology): “In contrast, ensemble learning methods (You et al., 2017; Furlanello et al., 2018; Anil et al., 2018; Dvornik et al., 2019; Park & Kwak, 2019; Liu et al., 2019; Wu et al., 2019) allow to combine multiple heterogeneous weak classifiers by averaging the predictions of the individual models instead.”
Incremental Few-Shot Object Detection (2020, neutral): “Note that there are many more recent FSL methods [6, 26, 7, 17, 47, 55, 50, 27, 39, 11, 16] that produce better performance on image classification tasks than MAML.”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, background): “It is worth mentioning that the amount of learnable parameters in SIB+EBM is around 80% less than that of model in [12] which ensembles 5 deep networks in parallel (and later learns a distillation network).”
Meta-Learning with Shared Amortized Variational Inference (2020, unknown): “…approach to handle small training sets is to use nearest neighbor (Wein-berger et al., 2006; Guillaumin et al., 2009; Vinyals et al., 2016), or nearest prototype (Mensink et al., 2012; Snell et al., 2017; Dvornik et al., 2019; Ren et al., 2018; Oreshkin et al., 2018) classiﬁcation methods.”
Adversarial Feature Hallucination Networks for Few-Shot Learning (2020, methodology): “It beats the best metric learning based method DCEM [8] by about 3.”
Progressive Meta-Learning With Curriculum (2022, background): “It is well known that integrating diverse predictions of different networks can significantly reduce prediction variance, and then achieve enhanced generalization performance [39], [40].”
Curriculum-Based Meta-learning (2021, background): “It has been demonstrated that integrating the predictions of diverse networks can significantly reduce the variance of classifiers, as a consequence, achieve improved generalization performance [8, 50].”
A New Benchmark for Evaluation of Cross-Domain Few-Shot Learning (2019, unknown): “Unlike previous works on ensemble methods for few-shot learning [7] that 79.57% ± 0.51% 87.67% ± 0.46% 90.93% ± 0.45% 88.07% ± 0.56% 95.68% ± 0.76% 97.64% ± 0.59% Ft Last-3 78.04% ± 0.77% 87.52% ± 0.53% 90.83% ± 0.42% 89.11% ± 0.47% 95.31% ± 0.85% 97.45% ± 0.46% Transductive Ft 81.76% ± 0.48%…”
Optimized Generic Feature Learning for Few-shot Classification across Domains (2020, methodology): “Recently, it has been shown that the standard feature learning approach, of first training a backbone network on a large number of base classes and learning a few-shot classifier on top, is competitive [2, 11, 54].”
Baby steps towards few-shot learning with multiple semantics (2019, unknown): “ensemble [8] 59.”
Deep Ensembles for Low-Data Transfer Learning (2020, background): “For few-shot classification on a new class, Dvornik et al. (2019) construct ensembles of mean-centroid classifiers from pre-trained ResNet18s.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “As a result, many new KD methods [9], [51], [56], [57], [80], [101], [114], [118], [131], [134], [139], [147], [155], [161], [174], [175], [182], [185], [186], [191], [195], [196], [201], [205], [208], [210], [214], [219], [231], [232], [234], [249], [252], [258], [263], [267], [272], [275] have bee”