A New Benchmark for Evaluation of Cross-Domain Few-Shot Learning

By Yunhui Guo, Noel C. F. Codella, Leonid Karlinsky, John R. Smith, Tajana Šimunić, Rogério Schmidt Feris. 2019

45 citations recorded by the source corpus.

Recent progress on few-shot learning has largely re-lied on annotated data for meta-learning, sampled from the same domain as the novel classes. However, in many applications, collecting data for meta-learning is infeasible or impossible. This leads to the cross-domain few-shot learn-ing problem, where a large domain shift exists between base and novel classes. Although some preliminary investigation of the few-shot methods under domain shift exists, a standard benchmark for cross-domain few-shot learning is not yet established. In this paper, we propose the cross-domain few-shot learning (CD-FSL) benchmark, consist-ing of images from diverse domains with varying similarity to ImageNet, ranging from crop disease images, satellite images, and medical images. Extensive experiments on the proposed benchmark are performed to compare an array of state-of-art meta-learning and transfer learning approaches, including various forms of single model fine-tuning and ensemble learning. The results demonstrate that current meta-learning methods underperform in relation to simple fine-tuning by 12.8% average accuracy. Accuracy of all methods tend to correlate with dataset similarity toImageNet. In addition, the relative performance gain with increasing number of shots is greater with transfer methods compared to meta-learning. Finally, we demonstrate that transferring from multiple pretrained models achieves best performance, with accuracy improvements of 14.9% and 1.9% versus the best of meta-learning and single model fine-tuning approaches, respectively. In summary, the proposed benchmark serves as a challenging platform to guide future research on cross-domain few-shot learning due to its spectrum of diversity and coverage

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Meta & Continual Learning

Citations in the corpus

Florence: A New Foundation Model for Computer Vision (2021, neutral): “The Cross-Domain Few-Shot learning benchmark (Guo et al., 2020) is used to measure an algorithm’s capability to adapt to downstream few-shot target tasks, containing domains with varying levels of dissimilarity to typical consumer photographs.”
Meta-Learning in Neural Networks: A Survey (2020, background): “ble in many potential practical applications of meta-learning, for example generalizing few-shot visual learning from everyday training images of ImageNet to specialist domains such as medical images [221]. From the perspective of a learner, this is a meta-level generalization of the domain-shift pr”
Improved Schemes for Episodic Memory-based Lifelong Learning (2019, background): “Inspired by few-shot learning [48, 49, 50, 51], in this section we consider a more challenging setting for lifelong learning where each task only has a limited number of examples.”
Feature Transformation Ensemble Model with Batch Spectral Regularization for Cross-Domain Few-Shot Classification (2020, methodology): “We investigate a number of variants of the proposed model by comparing with the strong ﬁne-tuning baseline result reported in [3].”