Uncertainty in Multitask Transfer Learning

By Alexandre Lacoste, Boris N. Oreshkin, Wonchang Chung, Thomas Boquet, Negar Rostamzadeh, David Krueger. 2018

21 citations recorded by the source corpus.

Using variational Bayes neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks. The result is a rich and meaningful prior capable of few-shot learning on new tasks. The posterior can go beyond the mean field approximation and yields good uncertainty on the performed experiments. Analysis on toy tasks shows that it can learn from significantly different tasks while finding similarities among them. Experiments of Mini-Imagenet yields the new state of the art with 74.5% accuracy on 5 shot learning. Finally, we provide experiments showing that other existing methods can fail to perform well in different benchmarks.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

Meta-Learning Probabilistic Inference for Prediction (2018, background): “Approaches that employ pre-training and/or residual networks (Bauer et al., 2017; Qiao et al., 2017; Rusu et al., 2018; Gidaris and Komodakis, 2018; Oreshkin et al., 2018; Garcia and Bruna, 2017; Lacoste et al., 2018) have been excluded so that the quality of the learning algorithm can be assessed s”
Meta-Learning with Latent Embedding Optimization (2018, background): “Other concurrent works have introduced deep parameter generators (Lacoste et al., 2018; Wu et al., 2018) that can better capture a wider distribution of model parameters, but do not employ gradient-based adaptation.”
Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning (2020, methodology): “Synbols In this setup, the models are pre-trained to classify characters from different alphabets on randomized backgrounds (Lacoste et al., 2018).”
Meta-Learning with Warped Gradient Descent (2019, methodology): “Our few-shot and supervised learning experiments take this approach Section 4.1; (c) Meta-learned prior : we can use meta-learners that deﬁne a full prior (Rusu et al., 2019; Oreshkin et al., 2018; Lacoste et al., 2018; Kim et al., 2018), enabling Bayesian approaches to WarpGrad.”
Evaluation Gaps in Machine Learning Practice (2022, background): “An exception to this trend was CV papers which explored shared representations (e.g., in multi-task learning [53, 99] or domain adaptation [119, 126]).”
Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels (2019, background): “As seen in [35], an effective prior learned from multiple tasks can be a powerful tool for learning new, unrelated tasks.”