A PAC-Bayesian bound for Lifelong Learning

By Anastasia Pentina, Christoph H. Lampert. 2013

222 citations recorded by the source corpus.

Transfer learning has received a lot of attention in the machine learning community over the last years, and several effective algorithms have been developed. However, relatively little is known about their theoretical properties, especially in the setting of lifelong learning, where the goal is to transfer information to tasks for which no data have been observed so far. In this work we study lifelong learning from a theoretical perspective. Our main result is a PAC-Bayesian generalization bound that offers a unified view on existing paradigms for transfer learning, such as the transfer of parameters or the transfer of low-dimensional representations. We also use the bound to derive two principled lifelong learning algorithms, and we show that these yield results comparable with existing methods.

Subfields: Artificial Intelligence, Computational Mechanics, Meta & Continual Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Meta & Continual Learning

In Statistical Learning Theory

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “2(d)), which is further validated by a theoretical analysis of the forgetting-generalization trade-off [94] and the PAC-Bayes bound of generalization errors [69], [95].”
Transductive Multi-View Zero-Shot Learning (2015, background): “Inspired by humans’ ability to recognise without seeing samples, and motivated by the prohibitive cost of training sample collection and annotation, the research area of learning to learn or lifelong learning [35], [6] has received increasing interests.”
Learnability and Algorithm for Continual Learning (2023, background): “Pentina & Lampert (2014) proposes a PAC-Bayesian framework to provide a learning bound on expected error by the average loss on the observed tasks.”
A Theoretical Study on Solving Continual Learning (2022, background): “[10] proposes a PAC-Bayesian framework to provide a learning bound on expected error in future tasks by the average loss on the observed tasks.”
Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, background): “Understanding its generalization performance is still limited (Pentina & Lampert, 2014; Bennani et al., 2020; Lee et al., 2021).”
Learngene: From Open-World to Your Learning Task (2021, methodology): “Here, we present the theoretical analysis for transfering representation from collective model and get the following bound based on PAC-Bayesian (Pentina and Lampert 2014) Theorem 1.”
Layerwise Optimization by Gradient Decomposition for Continual Learning (2021, neutral): “Continual learning has been a long-standing research problem in the ﬁeld of machine learning [32, 45, 36, 47].”
Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, background): “[42] studied incremental task learning from a theoretical perspective with the goal of transferring knowledge to future tasks.”
Learning Cumulatively to Become More Knowledgeable (2016, background): “Our work is related to lifelong machine learning [5, 26, 35, 39].”
Recent Advances in Open Set Recognition: A Survey (2018, background): “To meet this challenge, several related research topics have been explored including lifelong learning [1], [2], transfer learning”
Continual Representation Learning for Biometric Identification (2020, neutral): “Continual learning is also named life-long learning [46, 55, 57], incremental learning [7, 48, 54] and sequential learning [5, 10] in previous works.”
Meta-Learning without Memorization (2019, background): “Pentina & Lampert (2014) extends the single task PAC-Bayes bound (McAllester, 1999) to the multi-task setting, which quantiﬁes the gap between empirical error on training tasks and the expected error on new tasks.”
Contextual Transformation Networks for Online Continual Learning (2021, background): “To achieve this goal, we ﬁrst revisit a popular result in learning multiple tasks that each task’s features are centered around a common vector (Evgeniou & Pontil, 2004; Aytar & Zisserman, 2011; Pentina & Lampert, 2014; Liu et al., 2019b).”