Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks

By Haruka Asanuma, S. Takagi, Yoshihiro Nagano, Yuki Yoshida, Y. Igarashi, M. Okada. 2021

26 citations recorded by the source corpus.

When a computational system continuously learns from an ever-changing environment, it rapidly forgets its past experiences. This phenomenon is called catastrophic forgetting. While a line of studies has been proposed with respect to avoiding catastrophic forgetting, most of the methods are based on intuitive insights into the phenomenon, and their performances have been evaluated by numerical experiments using benchmark datasets. Therefore, in this study, we provide the theoretical framework for analyzing catastrophic forgetting by using teacher-student learning. Teacher-student learning is a framework in which we introduce two neural networks: one neural network is a target function in supervised learning, and the other is a learning neural network. To analyze continual learning in the teacher-student framework, we introduce the similarity of the input distribution and the input-output relationship of the target functions as the similarity of tasks. In this theoretical framework, we also provide a qualitative understanding of how a single-layer linear learning neural network forgets tasks. Based on the analysis, we find that the network can avoid catastrophic forgetting when the similarity among input distributions is small and that of the input-output relationship of the target functions is large. The analysis also suggests that a system often exhibits a characteristic phenomenon called overshoot, which means that even if the learning network has once undergone catastrophic forgetting, it is possible that the network may perform reasonably well after further learning of the current task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “(2021) and Asanuma et al. (2021) delivered under this teacher-student assumption are now limited to only two tasks. Yet another task relationship is that every objective Lt is the composition of two functions ft ◦ g, where the function g : R → R is referred to as the shared representation; see, e.g.”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “While this article went to press, a preprint by Asanuma et al. (2021) appeared which analyses continual learning in a teacher-student setup for linear regression.”
How catastrophic can catastrophic forgetting be in linear regression? (2022, background): “Despite recent advances in theoretical understanding of continual learning [4, 6, 14, 32, 34], catastrophic forgetting is not fully understood even in simple models.”
Continual Learning in Linear Classification on Separable Data (2023, background): “…there are still very few analytic results on it, since most of the continual learning theory thus far has focused on regression settings (e.g., Bennani et al. (2020); Doan et al. (2021); Asanuma et al. (2021); Lee et al. (2021); Evron et al. (2022); Goldfarb & Hand (2023); Li et al. (2023)).”