Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent

By Mehdi Abbana Bennani, Masashi Sugiyama. 2020

85 citations recorded by the source corpus.

In continual learning settings, deep neural networks are prone to catastrophic forgetting. Orthogonal Gradient Descent (Farajtabar et al., 2019) achieves state-of-the-art results in practice for continual learning, although no theoretical guarantees have been proven yet. We derive the first generalisation guarantees for the algorithm OGD for continual learning, for overparameterized neural networks. We find that OGD is only provably robust to catastrophic forgetting across a single task. We propose OGD+, prove that it is robust to catastrophic forgetting across an arbitrary number of tasks, and that it verifies tighter generalisation bounds. The experiments show that OGD+ outperforms OGD on settings with long range memory dependencies, even though the models are not overparameterized. Also, we derive a closed form expression of the learned models through tasks, as a recursive kernel regression relation, which captures the transferability of knowledge through tasks. Finally, we quantify theoretically the impact of task ordering on the generalisation error, which highlights the importance of the curriculum for lifelong learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

In Statistical Learning Theory

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Sometheoreticalstudieswiththeneuraltangent kernel (NTK) [96], [97], [98], [99] suggest that an increase in task similarity may lead to more forgetting.”
Learnability and Algorithm for Continual Learning (2023, methodology): “Bennani et al. (2020) showed that a specific method called orthogonal gradient descent (OGD) gives a tighter generalization bound than SGD.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “We prove Proposition 6 in Appendix C, where we also review the OGD method; see Bennani et al. (2020); Doan et al. (2021) for different theoretical aspects of OGD. Since regression can be viewed as a single-layer linear network with a least-squares loss, formula (9) can be extended into deep continua”
A Theoretical Study on Solving Continual Learning (2022, background): “[67] shows that orthogonal gradient descent gives a tighter generalization bound than SGD.”
Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, methodology): “The trained model within the NTK regime is then given as follows (Bennani et al., 2020; Doan et al., 2021): (5) We omit the index c because each head is updated independently.”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “Bennani & Sugiyama (2020) gave guarantees on the error achieved with orthogonal gradient descent in the same regime.”
A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix (2020, methodology): “Under that framework, [Bennani et al., 2020] show that CL models can be expressed as a recursive kernel regression and prove generalization and performance guarantee of OGD under infinite memory setting.”
TAG: Task-based Accumulated Gradients for Lifelong learning (2021, background): “As a result, OGD often faces memory issues during run-time depending upon the size of the model and the subset [5].”
Provable Lifelong Learning of Representations (2021, methodology): “[Bennani et al., 2020, Doan et al., 2021] analyze the generalization of the OGD algorithm [Farajtabar et al., 2020] through NTK [Jacot et al., 2018].”
Continual Learning in Linear Classification on Separable Data (2023, background): “…there are still very few analytic results on it, since most of the continual learning theory thus far has focused on regression settings (e.g., Bennani et al. (2020); Doan et al. (2021); Asanuma et al. (2021); Lee et al. (2021); Evron et al. (2022); Goldfarb & Hand (2023); Li et al. (2023)).”
Provable Continual Learning via Sketched Jacobian Approximations (2021, unknown): “Some recent theoretical developments include (Yin et al., 2020; Bennani et al., 2020; Alquier et al., 2017; Doan et al., 2021).”
SketchOGD: Memory-Efficient Continual Learning (2023, unknown): “Under the Neural Tangent Kernel (NTK) regime, Bennani et al. [36] and Doan et al. [31] interpret OGD as a recursive kernel regression and analyze its forgetting behavior.”
Provable and Efficient Continual Representation Learning (2022, background): “However, the theoretical principles of continual learning is relatively less understood and the progress is under-whelming compared to the algorithmic advances despite recent progress (see [8, 17, 45]).”
How catastrophic can catastrophic forgetting be in linear regression? (2022, background): “Despite recent advances in theoretical understanding of continual learning [4, 6, 14, 32, 34], catastrophic forgetting is not fully understood even in simple models.”
Formalizing the Generalization-Forgetting Trade-off in Continual Learning (2021, background): “[5] provided generalization (in the context of training to test data generalization) guarantees with orthogonal gradient descent, and Yin et al.”