Dark Experience for General Continual Learning: a Strong, Simple Baseline
By Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, Simone Calderara. 2020
1,381 citations recorded by the source corpus.
Neural networks struggle to learn continuously, as they forget the old knowledge catastrophically whenever the data distribution changes over time. Recently, Continual Learning has inspired a plethora of approaches and evaluation settings; however, the majority of them overlooks the properties of a practical scenario, where the data stream cannot be shaped as a sequence of tasks and offline training is not viable. We work towards General Continual Learning (GCL), where task boundaries blur and the domain and class distributions shift either gradually or suddenly. We address it through Dark Experience Replay, namely matching the network's logits sampled throughout the optimization trajectory, thus promoting consistency with its past. By conducting an extensive analysis on top of standard benchmarks, we show that such a seemingly simple baseline outperforms consolidated approaches and leverages limited resources. To provide a better understanding, we further introduce MNIST-360, a novel GCL evaluation setting.
Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Radiology, Nuclear Medicine and Imaging
In Meta & Continual Learning
Citations in the corpus
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Besides, other realistic consider-ationshavebeenincorporated,suchasmultiplelabels [29],noisy labels [30], blurred boundary [31], [32], hierarchical granularity [33] and sub-populations [34], mixture of task similarity [35], long-tailed distribution [36], novel class discovery [37], [38],…”
- Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): ““Dark experience replay” (Buzzega et al., 2020) extends this method to work in a setting where task boundaries are not clearly defined.”
- Learnability and Algorithm for Continual Learning (2023, methodology): “For the replay-based methods, we follow the budget sizes of (Rebuffi et al., 2017; Buzzega et al., 2020).”
- The Ideal Continual Learner: An Agent That Never Forgets (2023, neutral): “…methods (Robins, 1993; Shin et al., 2017; Rebuffi et al., 2017; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019; Aljundi et al., 2019b; Buzzega et al., 2020; Verwimp et al., 2021; Bang et al., 2021; Kim et al., 2022; Zhang et al., 2022; Wang et al., 2022a; Saha & Roy, 2023),…”
- Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, methodology): “Function Distance Regularization (FDR; Benjamin et al. (2018)), Dark Experience Replay (DER++; Buzzega et al. (2020)) and CLS-ER (Arani et al., 2022) leverage soft targets in addition to ground truth labels to enforce consistency regularization across previous and current model predictions.”
- A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “Replay-based methods [9, 15, 32, 34, 39] correspond to the second group, and they store a few samples from previous tasks or generate them in order to rehearse knowledge during the training phases for subsequent tasks.”
- CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, methodology): “We also compare to three rehearsalbased methods to provide additional context to our results: Experience Replay [11], Bias Correction (BiC) [67], and Dark-Experience Replay++ (DERP++) [7].”
- A Theoretical Study on Solving Continual Learning (2022, methodology): “For the backbone structure, we follow [4, 26, 34].”
- On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “However, recent works put an increasing focus on the harder Class-Incremental setting (CIL) [4, 32, 15, 81], which entails the production of a unified prediction encompassing all seen classes.”
- Continual Learning by Modeling Intra-Class Variation (2022, methodology): “The batch size is set as 32 and the learning rate and learning rate decay schedule are the same as the DER (Buzzega et al., 2020).”
- Continual Training of Language Models for Few-Shot Learning (2022, methodology): “(13) DER++ (Buzzega et al., 2020) is a recent replay method using distillation to regularize the new task training.”
- Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning (2022, methodology): “…hyperparameter α in the gradient of Eq 2 to deal with the biased ER loss by balancing the weight of the memory loss and incoming loss; 2) Distillation-based memory loss : DER [Buzzega et al., 2020] employs the logits-based distillation loss for memory samples, instead of the cross entropy loss.”
- Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “To evaluate the effectiveness of ORDER for mitigating forgetting in SETS, we compare to SOTA CL baselines, including Experience replay (ER) [15], A-GEM[14], MER [48], GPM [49] and DEGCL [13].”
- DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, background): “Method Task-IL Class-IL 0 LwF [24] 92.16 - EWC [18] 77.44 - OSLA [39] 81.03 - DLCFT (Ours) 95.79 - 500 ER [38] 79.14 43.52 DER [4] 91.47 58.07 DER++ [4] 91.56 53.29 DLCFT (Ours) - 59.98”
- S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “…in Table 4 justifies the necessity of using the proposed growing classifier pool and independent 74.92 ± 0.25* BiC [67] 79.28 ± 0.30* DER++ [4] 79.70 ± 0.44* Co 2 L [6] 79.75 ± 0.84* DyTox [14] 79.21 ± 0.10 L2P [66] 81.07 ± 0.13* EWC [32] 0/class 49.19 SimCLR [10]-CaSSLe [18] 0/class 44.2*…”
- Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “We use ResNet-18 [11] as the backbone [4,6,10,11].”
- Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “DER++ [9] combines rehearsal with distillation loss [36,83,47] to retrain past experience and obtain state-of-the-art performance.”
- Incremental Task Learning with Incremental Rank Updates (2022, methodology): “We also present comparisons with some recent methods: IBP-WF [22], DER [5] and Adam-NSCL [37], in terms of average accuracy for one experiment on two datasets.”