Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory

By Ron Amit, R. Meir. 2017

186 citations recorded by the source corpus.

In meta-learning an agent extracts knowledge from observed tasks, aiming to facilitate learning of novel future tasks. Under the assumption that future tasks are 'related' to previous tasks, the accumulated knowledge should be learned in a way which captures the common structure across learned tasks, while allowing the learner sufficient flexibility to adapt to novel aspects of new tasks. We present a framework for meta-learning that is based on generalization error bounds, allowing us to extend various PAC-Bayes bounds to meta-learning. Learning takes place through the construction of a distribution over hypotheses based on the observed tasks, and its utilization for learning a new task. Thus, prior knowledge is incorporated through setting an experience-dependent prior for novel tasks. We develop a gradient-based algorithm which minimizes an objective function derived from the bounds and demonstrate its effectiveness numerically with deep neural networks. In addition to establishing the improved performance available through meta-learning, we demonstrate the intuitive way by which prior information is manifested at different levels of the network.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Adaptive Gradient-Based Meta-Learning Methods (2019, methodology): “Several works have built upon this theory for modern LTL, such as via a PAC-Bayesian perspective [3] or by learning the kernel for the ridge regression [18].”
Generalizing from a Few Examples (2019, methodology): “es the sample complexity, which is proved for methods that utilize unlabeled data set [12, 107], multitask learning [85], generative modeling [39], refining existing parameters [76] and meta-learning [4]. However, the analysis is lacked for recent methods such as conditional embedding learning and l”
Invariant Causal Prediction for Block MDPs (2020, background): “Recent prior works (Amit and Meir, 2018; Yin et al., 2019) have developed generalization bounds for the multi-task problem, but they depend on the number of tasks seen at training time, which can be prohibitively expensive given how sample inefﬁcient RL is even in the single task regime.”
Meta-Learning without Memorization (2019, methodology): “Closely following the arguments in (Amit & Meir, 2018), we apply a standard PAC-Bayes bound to each of these and combine the results with a union bound, resulting in the following Theorem.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “[278], Amit and Meir [279], Bhattacharyya et al.”
Empirical Bayes Transductive Meta-Learning with Synthetic Gradients (2020, methodology): “The idea of using HB or EB for meta-learning is not new: Amit & Meir (2018) derive an objective similar to that of HB using PAC-Bayesian analysis; Grant et al. (2018) show that MAML (Finn et al., 2017) can be understood as a EB method; Ravi & Beatson (2018) consider a HB extension to MAML and…”
Provable Guarantees for Gradient-Based Meta-Learning (2019, background): “Recently, Amit & Meir (2018) showed PAC-Bayesian generalization bounds for this setting, although without implying an efﬁcient algorithm.”
User-friendly introduction to PAC-Bayes bounds (2021, background): “) will play an important role in the study of deep learning (in the wake of [70]), in RL [197] and in meta-learning [155, 12, 169].”
Towards Knowledgeable Supervised Lifelong Learning Systems (2020, unknown): “…be easily adapted for new tasks (Santoro, Bartunov, Botvinick, Wierstra, & Lillicrap, 2016; Finn, Abbeel, & Levine, 2017), Bayesian-based methods (Amit & Meir, 2018; Yoon, Kim, Dia, Kim, Bengio, & Ahn, 2018), and methods based on finding low-dimensional latent spaces of features that are highly…”
Transformers Can Do Bayesian Inference (2021, methodology): “We presented Prior-Fitted Networks (PFNs), a novel way to efﬁciently approximate the posterior predictive distribution using deep neural networks and showed its capability on a set of diverse tasks and priors.”