Meta-Learning Probabilistic Inference for Prediction

By Jonathan Gordon, John F. Bronskill, Matthias Bauer, Sebastian Nowozin, Richard E. Turner. 2018

289 citations recorded by the source corpus.

This paper introduces a new framework for data efficient and versatile learning. Specifically: 1) We develop ML-PIP, a general framework for Meta-Learning approximate Probabilistic Inference for Prediction. ML-PIP extends existing probabilistic interpretations of meta-learning to cover a broad class of methods. 2) We introduce VERSA, an instance of the framework employing a flexible and versatile amortization network that takes few-shot learning datasets as inputs, with arbitrary numbers of shots, and outputs a distribution over task-specific parameters in a single forward pass. VERSA substitutes optimization at test time with forward passes through inference networks, amortizing the cost of inference and relieving the need for second derivatives during training. 3) We evaluate VERSA on benchmark datasets where the method sets new state-of-the-art results, handles arbitrary numbers of shots, and for classification, arbitrary numbers of classes at train and test time. The power of the approach is then demonstrated through a challenging few-shot ShapeNet view reconstruction task.

Subfields: Probabilistic & Bayesian Methods, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

In Meta & Continual Learning

Citations in the corpus

Foundation Posteriors for Approximate Probabilistic Inference (2022, neutral): “Meta-Inference Meta-learning has been used to do inference amortized over both queries and a family of generative models [16, 30, 36].”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, methodology): “Meta-learning algorithms which “learn to learn” [45, 9, 49, 11] hold partic(a) Frames from clean videos”
Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes (2019, methodology): “Similarly, our work can be viewed as a deterministic limit of ML-PIP [5] which employs a distributional treatment of the local-parameters ψ.”
Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML (2019, methodology): “Other optimization-based algorithms have also since been developed, for example [18, 10, 4, 17, 38], which learn functions to embed the support set and test examples of a few-shot learning task, and then learn the weights of a task-specific classifier (using the support set) to perform few-shot clas”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “For supervised learning, Rusu et al. (2019); Gordon et al. (2019); Finn et al. (2018) adapt model predictions using probabilistic latent task variables inferred via amortized approximate inference.”
Defining Benchmarks for Continual Few-Shot Learning (2020, methodology): “One example are Bayesian approaches, like those based on amortized networks (Gordon et al., 2019), hierarchical models (Grant et al.”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “, via synthesizing only lightweight adapter layers in the feature extractor [108], or class-wise classifier weight synthesis [45]).”
Generalizing from a Few Examples (2019, methodology): “These include object recognition [35, 36, 82], font style transfer [7], phrase grounding [162], image retrieval [130], object tracking [14], specific object counting in images [162], scene location recognition [74], gesture recognition [102], part labeling [24], image generation [34, 76, 107, 109], ”
Transformers Can Do Bayesian Inference (2021, background): “Extending standard loss derviations Gordon et al. (2019) to the meta-level, we can show that minimizing the Prior-Data NLL ℓ θ yields an approximation of the PPD in terms of cross-entropy and thus KL -Divergence: Insight 1.”
Reconciling meta-learning and continual learning with online mixtures of tasks (2018, background): “2, we compare with three recent meta-learning algorithms that report improved performance on the standard miniImageNet benchmark of Section 3, but are not explicitly posed to address the continual learning setting of evolving tasks: MT-net [30], PLATIPUS [15], and VERSA [20].”
CAML: Fast Context Adaptation via Meta-Learning (2018, methodology): “l et al., 2017) 46:61 0:78% 65:77 0:70% Meta-SGD (Li et al., 2017) 50:47 1:87% 64:03 0:94% REPTILE (Nichol and Schulman, 2018) 49:97 0:32% 65:99 0:58% MT-NET (Lee and Choi, 2018) 51:70 1:84% - VERSA (Gordon et al., 2018) 53:40 1:82% 67:37 0:86 MAML (32) (Finn et al., 2017a) 48:07 1:75% 63:15 0:91% M”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “we note that there also exist works exploring how to apply normalization to meta learning [61], [220], [221], reinforcement learning [65], [222], [223], unsupervised representation learning [224], [225], permutation-equivariant networks [226], [227], graph neural networks [228], ordinary differentia”
TaskNorm: Rethinking Batch Normalization for Meta-Learning (2020, methodology): “Another widely used class of meta-learners are amortized-inference based approaches e.g, V ERSA (Gordon et al., 2019) and CNAP S (Requeima et al., 2019a).”
Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels (2020, methodology): “, 2019) including recent hierarchical Bayesian methods (Gordon et al., 2019; Grant et al., 2018; Jerfel et al., 2019).”
Transformers learn in-context by gradient descent (2022, methodology): “learning a linear model on a meta-learned deep data representations (Finn et al., 2017; Finn & Levine, 2018; Gordon et al., 2019; Lee et al., 2019; Rusu et al., 2019; Raghu et al., 2020; von Oswald et al., 2021).”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, methodology): “Apart from (RL) methods mentioned above, related work in this direction can be found, a.o., in Garnelo et al. (2018); Gordon et al. (2019); Choi et al. (2019).”
Memory Efficient Meta-Learning with Large Images (2021, methodology): “CNAPS [4] and VERSA [17]), the aggregation of support set points is built in directly via a deep set encoder eφ1(·).”