Learning a Universal Template for Few-shot Dataset Generalization
By Eleni Triantafillou, Hugo Larochelle, Richard S. Zemel, Vincent Dumoulin. 2021
117 citations recorded by the source corpus.
Few-shot dataset generalization is a challenging variant of the well-studied few-shot classification problem where a diverse training set of several datasets is given, for the purpose of training an adaptable model that can then learn classes from new datasets using only a few examples. To this end, we propose to utilize the diverse training set to construct a universal template: a partial model that can define a wide array of dataset-specialized models, by plugging in appropriate components. For each new few-shot classification problem, our approach therefore only requires inferring a small number of parameters to insert into the universal template. We design a separate network that produces an initialization of those parameters for each given task, and we then fine-tune its proposed initialization via a few steps of gradient descent. Our approach is more parameter-efficient, scalable and adaptable compared to previous methods, and achieves the state-of-the-art on the challenging Meta-Dataset benchmark.
Subfields: Representation & Self-Supervised Learning, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Representation & Self-Supervised Learning
- Wei-Hong Li
- Xialei Liu
- Mohit Bansal
- Hakan Bilen
- Richard E. Turner
- Aliaksandra Shysheya
- Derek Tam
- Sebastian Nowozin
In Meta & Continual Learning
- Xialei Liu
- Hakan Bilen
- Wei-Hong Li
- Massimiliano Patacchiola
- Richard E. Turner
- John F. Bronskill
- Sebastian Nowozin
- Mohit Bansal
Citations in the corpus
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “[78] explore learning universal and dataset dependent parameters that can be blended for generalization.”
- Improving Task Adaptation for Cross-domain Few-shot Learning (2021, methodology): “To better analyze the results, we also report the average accuracy on seen domains, unseen domains, all domains and the average rank as in [22, 42].”
- FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification (2022, methodology): “Some of these approaches (e.g. TADAM (Oreshkin et al., 2018), FLUTE (Triantafillou et al., 2021), and Simple CNAPs (Bateni et al., 2020) use both a metric head and FiLM layers to adapt the backbone.”