Defining Benchmarks for Continual Few-Shot Learning

By Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal, Amos J. Storkey. 2020

42 citations recorded by the source corpus.

Both few-shot and continual learning have seen substantial progress in the last years due to the introduction of proper benchmarks. That being said, the field has still to frame a suite of benchmarks for the highly desirable setting of continual few-shot learning, where the learner is presented a number of few-shot tasks, one after the other, and then asked to perform well on a validation set stemming from all previously seen tasks. Continual few-shot learning has a small computational footprint and is thus an excellent setting for efficient investigation and experimentation. In this paper we first define a theoretical framework for continual few-shot learning, taking into account recent literature, then we propose a range of flexible benchmarks that unify the evaluation criteria and allows exploring the problem from multiple perspectives. As part of the benchmark, we introduce a compact variant of ImageNet, called SlimageNet64, which retains all original 1000 classes but only contains 200 instances of each one (a total of 200K data-points) downscaled to 64 x 64 pixels. We provide baselines for the proposed benchmarks using a number of popular few-shot learning algorithms, as a result, exposing previously unknown strengths and weaknesses of those algorithms in continual and data-limited settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

In Meta & Continual Learning

Citations in the corpus

Continual Training of Language Models for Few-Shot Learning (2022, background): “Several researchers have studied this problem recently (Antoniou et al., 2020; Qin and Joty, 2021; Jin et al., 2021; Xia et al., 2021; Yin et al., 2022).”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “CL [31] ✓ ✗ ✗ ✗ SSL [11] ✗ ✓ ✗ ✗ CFSL [6] ✓ ✗ ✓ ✗ SSCL [60] ✓ ✓ ✗ ✗ SSFSL [47] ✗ ✓ ✓ ✓ SETS (Ours) ✓ ✓ ✓ ✓ We summarize and compare different benchmarks in Table 1, including continual learning (CL), semi-supervised learning (SSL), continual few-shot learning (CFSL), semi-supervised continual learni”
CoMPS: Continual Meta Policy Search (2021, background): “Recent work has made progress toward using supervised meta-learning in an online setting (Ren et al., 2018; Finn et al., 2019; Khodak et al., 2019; Jerfel et al., 2019; Grant et al., 2019; Zhuang et al., 2019; Antoniou et al., 2020; Wallingford et al., 2020; Yu et al., 2020a; Yao et al., 2020).”
Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning (2021, background): “Antoniou et al. (2020) assumes the model visits a sequence of few-shot classification tasks while the test tasks consist of seen classes at training.”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “Some recent works have shown limitations of LLL benchmarks (Antoniou et al., 2020; Roady et al., 2020).”
IIRC: Incremental Implicitly-Refined Classification (2020, background): “Existing efforts for benchmarking the existing lifelong learning setups include CORe50 benchmark [25], and [4] that proposes a benchmark for continual few-shot learning.”
Continual Few-Shot Learning for Text Classification (2021, background): “The most closest to our work is from the vision community, where they proposed a benchmark suite containing few-shot datasets for continual learning and evaluation criteria (Antoniou et al., 2020).”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “There are a number of established FSL datasets that are used in this setting, such as miniImageNet [41], tieredIm-ageNet [204], SlimageNet [205], CUB-200 [110] and Om-niglot [86].”
Online Fast Adaptation and Knowledge Accumulation (OSAKA): a New Approach to Continual Learning (2020, background): “Continual-meta learning is now an active research field [47, 5].”
Learning to Learn and Remember Super Long Multi-Domain Task Sequence (2022, methodology): “Continual few-shot learning (CFSL) [8] is an application of CL to few-shot learning, usually within a single domain, and focuses on remembering previously learned few-shot tasks when learning on the current one.”
A large-scale benchmark for few-shot program induction and synthesis (2021, methodology): “Most of these methods assume that the input form is constant( (Iwata & Kumagai, 2020) being a recent exception) and few-shot learning datasets are mainly image classification (Lake et al., 2015; Vinyals et al., 2016; Ren et al., 2018; Antoniou et al., 2020; Chen et al., 2019; Triantafillou et al., 2”
Meta Learning on a Sequence of Imbalanced Domains with Difficulty Awareness (2021, background): “Continual few-shot learning [8] (CFSL) focuses on remembering previously learned few-shot tasks in a single domain.”
Wandering Within a World: Online Contextualized Few-Shot Learning (2020, background): “FSL [45] MOCA [15] Online Mixture [21] Online Meta [20] Continual FSL* [4] OSAKA* [5] OC-FSL (Ours) * denotes concurrent work.”
Variable-Shot Adaptation for Online Meta-Learning (2020, methodology): “Unlike prior online meta-learning works (Finn et al., 2019; Grant et al., 2019; Khodak et al., 2019; Zhuang et al., 2019; Ren et al., 2020; Antoniou et al., 2020; Wallingford et al., 2020; Yao et al., 2020), we develop an algorithm that is specifically tailored to leverage variable amounts of data, ”
Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels (2020, background): “Future work could focus on exploiting the ﬂexibility of the model in related settings, especially those merging continual and few-shot learning (Antoniou et al., 2020), where DKT has the potential to thrive.”