On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning

By Lorenzo Bonicelli, Matteo Boschini, Angelo Porrello, Concetto Spampinato, Simone Calderara. 2022

63 citations recorded by the source corpus.

Rehearsal approaches enjoy immense popularity with Continual Learning (CL) practitioners. These methods collect samples from previously encountered data distributions in a small memory buffer; subsequently, they repeatedly optimize on the latter to prevent catastrophic forgetting. This work draws attention to a hidden pitfall of this widespread practice: repeated optimization on a small pool of data inevitably leads to tight and unstable decision boundaries, which are a major hindrance to generalization. To address this issue, we propose Lipschitz-DrivEn Rehearsal (LiDER), a surrogate objective that induces smoothness in the backbone network by constraining its layer-wise Lipschitz constants w.r.t. replay examples. By means of extensive experiments, we show that applying LiDER delivers a stable performance gain to several state-of-the-art rehearsal CL methods across multiple datasets, both in the presence and absence of pre-training. Through additional ablative experiments, we highlight peculiar aspects of buffer overfitting in CL and better characterize the effect produced by LiDER. Code is available at https://github.com/aimagelab/LiDER

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Hang Su
5. Qiwen Wang
6. Jun Zhu
7. Ziwei Liu
8. Liyuan Wang

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Jun Zhu
4. Da-Wei Zhou
5. Hang Su
6. Ziwei Liu
7. Liyuan Wang
8. Qiwen Wang

In Meta & Continual Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Xingxing Zhang
5. Liyuan Wang
6. Hang Su
7. Qiwen Wang
8. Jun Zhu

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Many ideas for improving robustness to adversarial examples have been used to improve continual learning, such as flat minima [31], [81], model ensemble [69], Lipschitz continuity [157] and adversarial training [158].”
• The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “(2022); Bonicelli et al. (2022). For the first time to our knowledge, a theoretical justification that accounts for its effectiveness is provided here: Theorem 3 (Rehearsal ≈ ICL).”
• CaSpeR: Latent Spectral Regularization for Continual Learning (2023, unknown): “, 2022b) or deteriorating decision surface (Bonicelli et al., 2022).”
• Class-Incremental Learning: A Survey (2023, background): “…on a small pool of data inevitably leads to tight and unstable decision boundaries, several works tackle this problem by constraining the model’s layer-wise Lipschitz constants with regard to exemplars [163] or enlarging representational variations to alleviate representation collapse [164].”
• Consistent Prompting for Rehearsal-Free Continual Learning (2024, unknown): “Rehearsal-based methods [3, 34, 49] alleviate the forgetting problem with a small number of ex-emplars of previous tasks stored in the memory buffer and * indicates corresponding author. replayed with the new task data during training.”