Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting

By Ryo Karakida, Shotaro Akaho. 2021

16 citations recorded by the source corpus.

Sequential training from task to task is becoming one of the major objects in deep learning applications such as continual learning and transfer learning. Nevertheless, it remains unclear under what conditions the trained model's performance improves or deteriorates. To deepen our understanding of sequential training, this study provides a theoretical analysis of generalization performance in a solvable case of continual learning. We consider neural networks in the neural tangent kernel (NTK) regime that continually learn target functions from task to task, and investigate the generalization by using an established statistical mechanical analysis of kernel ridge-less regression. We first show characteristic transitions from positive to negative transfer. More similar targets above a specific critical value can achieve positive knowledge transfer for the subsequent task while catastrophic forgetting occurs even with very similar targets. Next, we investigate a variant of continual learning which supposes the same target function in multiple tasks. Even for the same target, the trained model shows some transfer and forgetting depending on the sample size of each task. We can guarantee that the generalization error monotonically decreases from task to task for equal sample sizes while unbalanced sample sizes deteriorate the generalization. We respectively refer to these improvement and deterioration as self-knowledge transfer and forgetting, and empirically confirm them in realistic training of deep neural networks as well.

Subfields: Artificial Intelligence, Meta & Continual Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bin Liu
2. Hang Su
3. Gyuhak Kim
4. Tatsuya Konishi
5. Jun Zhu
6. Changnan Xiao
7. Liyuan Wang
8. Xingxing Zhang

In Meta & Continual Learning

1. Liyuan Wang
2. Bin Liu
3. Xingxing Zhang
4. Gyuhak Kim
5. Hang Su
6. Jun Zhu
7. Tatsuya Konishi
8. Changnan Xiao

In Statistical Learning Theory

1. Tatsuya Konishi
2. Jun Zhu
3. Gyuhak Kim
4. Changnan Xiao
5. Bin Liu
6. Zixuan Ke

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Sometheoreticalstudieswiththeneuraltangent kernel (NTK) [96], [97], [98], [99] suggest that an increase in task similarity may lead to more forgetting.”
• A Theoretical Study on Solving Continual Learning (2022, background): “Instead of focusing on the traditional PAC generalization bound [10, 11], we focus on how to solve the CIL problem.”
• Open-World Continual Learning: Unifying Novelty Detection and Continual Learning (2023, background): “Instead of focusing on the traditional PAC generalization bound [48] or neural tangent kernel (NTK) [25], we focus on how to solve the CIL problem.”