How catastrophic can catastrophic forgetting be in linear regression?

By Itay Evron, Edward Moroshko, Rachel A. Ward, Nati Srebro, Daniel Soudry. 2022

90 citations recorded by the source corpus.

To better understand catastrophic forgetting, we study fitting an overparameterized linear model to a sequence of tasks with different input distributions. We analyze how much the model forgets the true labels of earlier tasks after training on subsequent tasks, obtaining exact expressions and bounds. We establish connections between continual learning in the linear setting and two other research areas: alternating projections and the Kaczmarz method. In specific settings, we highlight differences between forgetting and convergence to the offline solution as studied in those areas. In particular, when T tasks in d dimensions are presented cyclically for k iterations, we prove an upper bound of T^2 * min{1/sqrt(k), d/k} on the forgetting. This stands in contrast to the convergence to the offline solution, which can be arbitrarily slow according to existing alternating projection results. We further show that the T^2 factor can be lifted when tasks are presented in a random ordering.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “With t > 1, given (ŵt−1,Kt−1) and data (Xt,yt), one can compute (ŵt,Kt) via solving min w∈Kt−1 ‖Xtw − yt‖22 (7) ⇔min a ‖Xt(ŵt−1 +Kt−1a)− yt‖22 (8) In particular, (8) can be solved via SVD.3 Compared to the method of Evron et al. (2022), ICL† uses some extra storage for Kt to attain optimality and”
Continual Learning in Linear Classification on Separable Data (2023, methodology): “We thus focus on ∥wk+1 −wk∥ and bound it using all of the above, using ideas from Evron et al. (2022): ∥wk+1 −wk∥2 Lemma D.8 ≤ 1 k + 1 k∑ t=0 ∥wt −wt+1∥2 = 1 k + 1 k∑ t=0 ∥wt −wt+1 −w⋆ +w⋆∥2 = 1 k + 1 k∑ t=0 ∥(I−Pt+1) (wt)− (I−Pt+1) (w⋆)∥2 [Prop.”