Transfer without Forgetting

By Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Simone Palazzo, Concetto Spampinato, Simone Calderara. 2022

74 citations recorded by the source corpus.

This work investigates the entanglement between Continual Learning (CL) and Transfer Learning (TL). In particular, we shed light on the widespread application of network pretraining, highlighting that it is itself subject to catastrophic forgetting. Unfortunately, this issue leads to the under-exploitation of knowledge transfer during later tasks. On this ground, we propose Transfer without Forgetting (TwF), a hybrid approach building upon a fixed pretrained sibling network, which continuously propagates the knowledge inherent in the source domain through a layer-wise loss term. Our experiments indicate that TwF steadily outperforms other CL methods across a variety of settings, averaging a 4.81% gain in Class-Incremental accuracy over a variety of datasets and different buffer sizes.

Subfields: Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “TwF [200] also trains a sibling network, but distills knowledge from the backbone in a layer-wise manner.”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “To assess our proposal, we perform a suite of experiments through the Mammoth framework [12, 17, 9, 6, 27, 13, 14, 50], an open-source codebase introduced in [15] for testing CL algorithms.”
Online Continual Learning For Interactive Instruction Following Agents (2024, methodology): “In particular, distillation-based approaches (Li & Hoiem, 2017; Koh et al., 2023; Boschini et al., 2022b) have been extensively investigated to leverage prior knowledge, but often require substantial memory and additional computation.”
InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning (2024, background): “…of the subspace M ⊥ t in different Transformer layers of ViT-B/16. ﬁnd We can that the dimension of the subspace M ⊥ t in different Transformer layers of ViT-B/16 is always much larger than zero, which means the space for learning the new task always exists throughout the whole learning process.”
CaSpeR: Latent Spectral Regularization for Continual Learning (2023, background): “The use of pre-trained models [8, 47], exploiting also transformers architectures [44], has been showing increasing popularity in the recent CL literature.”