Continual Learning in Linear Classification on Separable Data

By Itay Evron, Edward Moroshko, G. Buzaglo, M. Khriesh, B. Marjieh, Nati Srebro, Daniel Soudry. 2023

33 citations recorded by the source corpus.

We analyze continual learning on a sequence of separable linear classification tasks with binary labels. We show theoretically that learning with weak regularization reduces to solving a sequential max-margin problem, corresponding to a special case of the Projection Onto Convex Sets (POCS) framework. We then develop upper bounds on the forgetting and other quantities of interest under various settings with recurring tasks, including cyclic and random orderings of tasks. We discuss several practical implications to popular training practices like regularization scheduling and weighting. We point out several theoretical differences between our continual classification setting and a recently studied continual regression setting.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. René Vidal
2. Paris V. Giampouras
3. Liangzu Peng

Citations in the corpus

• The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “5 of (Evron et al., 2023) (based on Fisher information) and Proposition 3 (based on the first-order optimality conditions).”