Continual Learning in the Teacher-Student Setup: Impact of Task Similarity

By Sebastian Lee, Sebastian Goldt, Andrew M. Saxe. 2021

103 citations recorded by the source corpus.

Continual learning-the ability to learn many tasks in sequence-is critical for artificial learning systems. Yet standard training methods for deep networks often suffer from catastrophic forgetting, where learning new tasks erases knowledge of earlier tasks. While catastrophic forgetting labels the problem, the theoretical reasons for interference between tasks remain unclear. Here, we attempt to narrow this gap between theory and practice by studying continual learning in the teacher-student setup. We extend previous analytical work on two-layer networks in the teacher-student setup to multiple teachers. Using each teacher to represent a different task, we investigate how the relationship between teachers affects the amount of forgetting and transfer exhibited by the student when the task switches. In line with recent work, we find that when tasks depend on similar features, intermediate task similarity leads to greatest forgetting. However, feature similarity is only one way in which tasks may be related. The teacher-student approach allows us to disentangle task similarity at the level of readouts (hidden-to-output weights) and features (input-to-hidden weights). We find a complex interplay between both types of similarity, initial transfer/forgetting rates, maximum transfer/forgetting, and long-term transfer/forgetting. Together, these results help illuminate the diverse factors contributing to catastrophic forgetting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Sometheoreticalstudieswiththeneuraltangent kernel (NTK) [96], [97], [98], [99] suggest that an increase in task similarity may lead to more forgetting.”
Learnability and Algorithm for Continual Learning (2023, background): “Lee et al. (2021) studied the generalization error by task similarity.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “It appears to us that the insights Lee et al. (2021) and Asanuma et al. (2021) delivered under this teacher-student assumption are now limited to only two tasks.”
A Theoretical Study on Solving Continual Learning (2022, background): “The work in [66] studies the generalization error by task similarity and [11] studies the dependence of generalization error on sample size or number of tasks including forward and backward transfer.”
Task-Free Continual Learning via Online Discrepancy Distance Learning (2022, neutral): “Recent attempts have provided the theoretical analysis for CL from different perspectives including the risk bound [46, 51], NP-hard problem [17], Teacher-Student framework [23, 58] and game theory [37].”
Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, background): “This is complementary to earlier studies on incremental learning of new classes and its catastrophic forgetting (Ramasesh et al., 2020; Lee et al., 2021), where each task includes different classes and does not allow the revisit.”
Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, background): “In recent years, some studies have provided the theoretical analysis for continual learning from different perspectives including the NP-hard problem [27], risk bound [58,64], Teacher-Student framework [34,57] and game theory [43].”
A Theory for Knowledge Transfer in Continual Learning (2022, background): “Theoretical studies in continual learning have studied the effects of task similarity in catastrophic forgetting (Lee et al., 2021), and discovered that optimal continual learning is NP-hard and requires perfect memory (Knoblauch et al., 2020).”
Continual Learning in Linear Classification on Separable Data (2023, background): “Many papers have investigated various aspects of catastrophic forgetting, including when it occurs (Evron et al., 2022), strategies to avoid it (Peng et al., 2023), the influence of task similarity (Lee et al., 2021), its impact on transferability (Chen et al., 2023), and other related factors.”
Provable and Efficient Continual Representation Learning (2022, background): “We also emphasize that there are several interesting works on the theory of continual learning such as Lee et al. (2021); Doan et al. (2021); Bennani et al. (2020); Mirzadeh et al. (2020); Yin et al. (2020).”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “Several recent works have investigated the role of task similarity for catastrophic forgetting (Nguyen et al., 2019; Ramasesh et al., 2021; Lee et al., 2021).”
How catastrophic can catastrophic forgetting be in linear regression? (2022, background): “Despite recent advances in theoretical understanding of continual learning [4, 6, 14, 32, 34], catastrophic forgetting is not fully understood even in simple models.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “Existing works have achieved this either via variational inference (Adel et al., 2020) or via expectation maximization (Lee et al., 2021a).”