Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

By Eleni Triantafillou, Tyler Lixuan Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, Hugo Larochelle. 2019

701 citations recorded by the source corpus.

Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking. To address this limitation, we propose Meta-Dataset: a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners on Meta-Dataset, along with a competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models' ability to leverage diverse training sources for improving their generalization. We also propose a new set of baselines for quantifying the benefit of meta-learning in Meta-Dataset. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

On Robustness and Transferability of Convolutional Neural Networks (2020, background): “Recent benchmarks have been proposed to evaluate transfer to several datasets, to assess generalization to tasks with different characteristics, or those disjoint from the pre-training data [62, 69].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Other supervised learning benchmarks focus on performance on multiple data sets, either via transfer learning, meta-learning, or multitask learning [46, 57].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Meta-Dataset (Triantaﬁllou et al., 2019) contains natural image classiﬁcation tasks.”
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “Meta-Dataset [50] is a new benchmark for evaluating few-shot methods in largescale settings.”
Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning (2021, methodology): “Tseng et al. (2020) used a randomly split manner for the cross-domain test and Triantafillou et al. (2020) used all traffic signs dataset for the test.”
True Few-Shot Learning with Language Models (2021, neutral): “The meta-learning setting assumes access to data from many other tasks in order to perform learning and model selection [20, 21, 64, 65].”
Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers (2021, methodology): “The ten datasets from Meta-Dataset (Triantaﬁllou et al., 2019) are used in our study: Aircraft, Bird, COCO, Describable Texture, Flower, Fungi, ImageNet, Omniglot, Quickdraw and Trafﬁc Sign.”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, methodology): “SUPER-NATURALINSTRUCTIONS is a metadataset (Triantafillou et al., 2019) consisting of a variety of NLP tasks (see Fig.”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, methodology): “Unlike existing datasets [39, 8, 26, 49, 48], ORBIT show objects in a wide range of realistic conditions, including when objects are poorly framed, occluded by hands and other objects, blurred, and in a wide variation of backgrounds, lighting, and object orientations.”
CrossTransformers: spatially-aware few-shot transfer (2020, methodology): “Note that in this table, QuickDraw uses the split from the original paper [84] rather than the (somewhat easier) split published for that paper’s public benchmark.”
Meta-learning with differentiable closed-form solvers (2018, background): “Also: Visual Decathlon[1] and Meta-dataset[2] span multiple domains.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “This idea of “learning to learn” has been widely explored in computer vision and robotics community (Yu et al., 2020; Triantafillou et al., 2020).”
Learngene: From Open-World to Your Learning Task (2021, methodology): “…architectures of eight representative methods ((Finn, Abbeel, and Levine 2017; Nichol, Achiam, and Schulman 2018; Snell, Swersky, and Zemel 2017; Triantaﬁllou et al. 2020; Vinyals et al. 2016; Zhang et al. 2020), two ﬁne-tuning methods of (Chen et al. 2019)) to match ours on the data of novel…”
Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML (2019, methodology): “This algorithm, with deep neural networks as the underlying model, has been highly influential, with significant follow on work [19, 6, 23, 11, 5, 9, 28, 31, 12].”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “There are three sub-challenges: (i) The ﬁrst challenge is ﬁtting a meta-learner to a wide distribution of tasks p ( T ) , which as we have seen is challenging for existing methods [206], [207], [230], and may be partly due to conﬂicting gradients between tasks [285].”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, methodology): “Number of shots Following prior work outside NLP [47, 67], our benchmark samples the training shot independently for each episode E and class i, as K i ∼ Unif(Kmin,Kmax), where Kmin = 1.”