Regularization Shortcomings for Continual Learning

By Timothée Lesort, Andrei Stoian, David Filliat. 2019

50 citations recorded by the source corpus.

In most machine learning algorithms, training data are assumed independent and identically distributed (iid). Otherwise, the algorithms' performances are challenged. A famous phenomenon with non-iid data distribution is known as \say{catastrophic forgetting}. Algorithms dealing with it are gathered in the \textit{Continual Learning} research field. In this article, we study the \textit{regularization} based approaches to continual learning. We show that those approaches can not learn to discriminate classes from different tasks in an elemental continual benchmark: class-incremental setting. We make theoretical reasoning to prove this shortcoming and illustrate it with examples and experiments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “…continual learning, including regularization-based methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Liu et al., 2018; Ritter et al., 2018; Lesort et al., 2019; Park et al., 2019; Yin et al., 2020; Zhou et al., 2021; Heckel, 2022), memory-based methods (Robins, 1993; Shin et al., 2017;…”
Constrained Few-shot Class-incremental Learning (2022, background): “Regularization strategies have been aimed at avoiding such forgetting [22], but they have been recently shown to be insufficient in CIL [30].”
New Insights on Reducing Abrupt Representation Change in Online Continual Learning (2021, background): “The single-head assumption is further studied in task-agnostic continual learning settings (He et al., 2019; Caccia et al., 2020; Ostapenko et al., 2021; Von Oswald et al., 2021; Lesort et al., 2019) in which the task-boundary assumption, amongst others, is also relaxed.”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “Regularization methods mostly protect the model’s ability to classify within a task, and thus they do not work well in our setting, which requires the ability to classify from all labels the model has seen before [34].”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Regularization [12, 11] is one of the directions but [111] shows that this approach has theoretical limitations in the class incremental setting and cannot be used alone to reach decent performance.”
Online Class-Incremental Continual Learning with Adversarial Shapley Value (2020, methodology): “…methods that utilize a generative model to produce virtual samples instead of using a memory buffer (Shin et al. 2017), recent research has demonstrated the limitations of such approaches with convolutional neural networks in datasets such as CIFAR-10 (Aljundi et al. 2019a; Lesort et al. 2019).”
Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning (2020, methodology): “Thus, task-agnostic (or task-free) CL has been motivated and studied in recent works (Aljundi et al., 2019b; Zeno et al., 2018; He et al., 2019; Lesort et al., 2019b).”
Adversarial Shapley Value Experience Replay for Task-Free Continual Learning (2020, methodology): “We focus on ER-based methods for comparison since recent research has demonstrated the shortcomings of prior-focused methods in the task-free setting [18, 9], and [1] has shown ER-based methods outperform other memory-based methods.”
Continual Learning in Deep Networks: an Analysis of the Last Layer (2021, background): “However, recently [31, 23] showed that regularization were not sufficient for incremental scenarios.”
Pretrained Language Model in Continual Learning: A Comparative Study (2022, methodology): “These methods are often referred to as generative replay Shin et al. (2017); Lesort et al. (2019a); Ostapenko et al. (2019).”
Online Continual Learning For Interactive Instruction Following Agents (2024, methodology): “(3) Regularization-Based Method (Zenke et al., 2017; Kirkpatrick et al., 2017; Lesort et al., 2019) prevents the overwriting of important parameters by imposing a penalty on changes to these crucial parameters.”
Understanding Continual Learning Settings with Data Distribution Drift Analysis (2021, background): “…evaluation in continual learning often referred to as “ﬁnal performance” (Chaudhry et al., 2019; Kirkpatrick et al., 2017; Shin et al., 2017; Lesort et al., 2019c), however in a more general scope, it evaluates the current performance of an algorithm on a test task at a particular point…”
Scaling the Number of Tasks in Continual Learning (2022, background): “Indeed, without replay to learn discriminative features between two classes, the classes need to be in the same task Lesort et al. [2019].”
Continual Learning with Deep Artificial Neurons (2020, background): “al, historical data distribution; thus requiring a non-iid assumption. Broadly speaking, 4 categories of techniques have emerged to combat catastrophic forgetting: (1) regularization-based approaches [4, 22, 24], (2) knowledge-compression, or capacity expansion [20, 23, 28, 38], (3) experience repla”