Efficient Continual Learning in Neural Networks with Embedding Regularization

By Jary Pomponi, Simone Scardapane, Vincenzo Lomonaco, Aurelio Uncini. 2019

47 citations recorded by the source corpus.

Continual learning of deep neural networks is a key requirement for scaling them up to more complex applicative scenarios and for achieving real lifelong learning of these architectures. Previous approaches to the problem have considered either the progressive increase in the size of the networks, or have tried to regularize the network behavior to equalize it with respect to previously observed tasks. In the latter case, it is essential to understand what type of information best represents this past behavior. Common techniques include regularizing the past outputs, gradients, or individual weights. In this work, we propose a new, relatively simple and efficient method to perform continual learning by regularizing instead the network internal embeddings. To make the approach scalable, we also propose a dynamic sampling strategy to reduce the memory footprint of the required external storage. We show that our method performs favorably with respect to state-of-the-art approaches in the literature, while requiring significantly less space in memory and computational time. In addition, inspired inspired by to recent works, we evaluate the impact of selecting a more flexible model for the activation functions inside the network, evaluating the impact of catastrophic forgetting on the activation functions themselves.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Online Continual Learning Via Candidates Voting (2021, methodology): “Replay-based methods are shown to be effective for maintaining learned knowledge by either using the original data as exemplars [35, 34, 29, 6, 27, 33, 2, 3, 8, 37, 9, 7] or synthetic data and statistics [38, 40, 43, 22].”
Latent Replay for Real-Time Continual Learning (2019, background): “proved to be an effective approach to contrast forgetting in continual learning scenarios [17], [18], [19], [20].”
The Effectiveness of Memory Replay in Large Scale Continual Learning (2020, methodology): “One recent work that is particularly related to our proposed approach is the embedding regularization algorithm (Pomponi et al., 2020).”
VariGrow: Variational Architecture Growing for Task-Agnostic Continual Learning based on Bayesian Novelty (2022, background): “…when learning new ones by penalizing drastic changes to a model’s parameters (Kirkpatrick et al., 2017; Aljundi et al., 2018; Titsias et al., 2019; Pomponi et al., 2020), and 3) expansion-based methods which grow and assign new model components for different tasks, keeping unrelated model…”
Task-agnostic Continual Learning with Hybrid Probabilistic Models (2021, unknown): “…regularization [Titsias et al., 2019]; feature regularization, and feature replay methods [Pomponi et al., 2020a, van de Ven et al., 2020, Pomponi et al., 2020b]; and orthogonality-based regularized replay methods such as OGD [Farajtabar et al., 2020], AGEM [Chaudhry et al., 2018] and…”
Practical Recommendations for Replay-based Continual Learning Methods (2022, methodology): “Selecting examples in GSS [2] consists of maximizing the diversity of samples in the replay buﬀer as suggested in [20] but using the gradient values.”
GROWN: GRow Only When Necessary for Continual Learning (2021, methodology): “, (Rebuffi et al. 2017; Pomponi et al. 2020)) based on the combination of the above strategies.”
SpaceNet: Make Free Space For Continual Learning (2020, methodology): “Some other methods combine the rehearsal and regularization strategies [33, 34].”
Unsupervised Continual Learning Via Pseudo Labels (2021, methodology): “Based on data type, there are pseudo rehearsal based methods [44, 48, 20, 52], which use synthetic or embeddings of data, and methods use original data as exemplars [39, 31, 8, 28, 36, 37, 2, 4, 10, 9].”