Latent Replay for Real-Time Continual Learning

By Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, Davide Maltoni. 2019

193 citations recorded by the source corpus.

Training deep neural networks at the edge on light computational devices, embedded systems and robotic platforms is nowadays very challenging. Continual learning techniques, where complex models are incrementally trained on small batches of new data, can make the learning problem tractable even for CPU-only embedded devices enabling remarkable levels of adaptiveness and autonomy. However, a number of practical problems need to be solved: catastrophic forgetting before anything else. In this paper we introduce an original technique named "Latent Replay" where, instead of storing a portion of past data in the input space, we store activations volumes at some intermediate layer. This can significantly reduce the computation and storage required by native rehearsal. To keep the representation stable and the stored activations valid we propose to slow-down learning at all the layers below the latent replay one, leaving the layers above free to learn at full pace. In our experiments we show that Latent Replay, combined with existing continual learning techniques, achieves state-of-the-art performance on complex video benchmarks such as CORe50 NICv2 (with nearly 400 small and highly non-i.i.d. batches) and OpenLORIS. Finally, we demonstrate the feasibility of nearly real-time continual learning on the edge through the deployment of the proposed technique on a smartphone device.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, positive): “Buffer size Average Acc ( ↑ ) Forgetting ( ↑ ) LRCIL [51] 100/class 76.39* -4.39* iCaRL [48] 79.76* -8.73* LUCIR [22] 82.53* -5.34* LRCIL [51] 50/class 74.01* -8.62* iCaRL [48] 73.98* -14.50* LUCIR [22] 80.77* -7.85* DyTox [14] 86.21 -1.55 EWC [32] 0/class is large (e.g., 345 classes in DomainNet).”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “A few other methods were also proposed to store either compressed data instances [96] or activation volumes [97].”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…in some recent works that replay high-level feature representations (Hayes, Kafle, Shrestha, Acharya, & Kanan, 2020; Iscen, Zhang, Lazebnik, & Schmid, 2020; Caccia, Belilovsky, Caccia, & Pineau, 2020; Pellegrini, Graffieti, Lomonaco, & Maltoni, 2019; van de Ven, Siegelmann, & Tolias, 2020).”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Since the model changes along the training process, to keep the latent features valid, [118] proposes to slow-down—in the limit case, freeze—the learning of all the layers before the feature extraction layer, while [65] proposes a feature adaptation method to map previous features to their correct v”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, methodology): “To reduce storage, it is advised to use either latent replay (Pellegrini et al., 2019) or pseudo-rehearsal (Robins, 1995) methods.”
Generative Feature Replay For Class-Incremental Learning (2020, background): “In [38], they propose to store activations for replay and a slow-down learning at all the layers below the replay layer.”
Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation (2022, methodology): “Existing work in LLL adopts the replay based methods (Pellegrini et al., 2019), such as Latent Replay, or regularization based methods (Huszár, 2018; Li and Hoiem, 2018), such as Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017), Synaptic Intelligence (Zenke et al., 2017).”
Avalanche: an End-to-End Library for Continual Learning (2021, neutral): “The Evaluator object directly interacting with the algorithm provides a uniﬁed interface to control and compute several performance metrics ( p i ), delegating results logging to the Logger(s) objects. to Robotics [28, 40, 38], from Reinforcement Learning [25, 31] to Sequence Learning [8], among oth”
CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions (2020, methodology): “• AR1* with Latent Replay : a recently proposed strategy (Pellegrini et al., 2020) showing competitive re-sults on CORe50 with a shared, non ﬁne-tuned hyper-parametrization across the three main tracks.”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “To this end low layers are either regularized (Liu et al., 2020), trained at a slower pace (Pellegrini et al., 2020), or, as. in (van de Ven et al., 2020; Hayes et al., 2020), frozen.”
A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays (2021, methodology): “Among these groups, rehearsal CL strategies have emerged as the most effective to deal with catastrophic forgetting, at the cost of an additional replay memory [1], [27], [28].”
Online Continual Learning for Embedded Devices (2022, background): “, 2019), video streams (Hayes et al., 2020; Hayes & Kanan, 2020; Pellegrini et al., 2020; 2021), imbalanced data streams (Aggarwal et al.”
Continuous Domain Adaptation with Variational Domain-Agnostic Feature Replay (2020, background): “A concurrent work (Pellegrini et al., 2019) that uses latent replay is closely related to our feature replay, both emphasizing on the high-level knowledge transfer, however, their latent replay stands for the replay of the activation volumes in some of the intermediate layers without stochasticity.”
Multi-Domain Multi-Task Rehearsal for Lifelong Learning (2020, methodology): “2019), the model can retrain the saved data with the training on the new task together, and these methods often construct single objective for the new task and set the saved data from old tasks as the constrains; (2) by saving the latent features for selected samples (Latent-rehearsal) (Pellegrini e”
A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials (2022, methodology): “From the results in Table 9, we can discover that LabelSM slightly improves the AA scores in the cases of iCaRL [60] and LUCIR [28], while hurting the AA scores in the case of LRCIL [59] and CCIL [55].”
Online Continual Learning on Sequences (2020, methodology): “Overall, AR1* combined with latent replay shows to be substantially more eﬀective for OCL on sequential data streams with a good resource consumption trade-oﬀ [67]).”
Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks (2021, background): “Latent Replay [16] Rehearsal MobileNetV1 • Fast learning • Low computational and memory requirements Low accuracy”