GDumb: A Simple Approach that Questions Our Progress in Continual Learning

By Ameya Prabhu, Philip H. S. Torr, Puneet Kumar Dokania. 2020

735 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “GDumb [159] greedily collects incoming training samples in a memory buffer and then uses them to train a model from scratch for testing.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “(18) Rehearsal has shown good empirical performance in deep continual learning; see, e.g., Chaudhry et al. (2019); Prabhu et al. (2020); Zhang et al. (2022); Bonicelli et al. (2022).”
First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, methodology): “We show that this approach is comparable to a softmax classifier in the offline setting and that it outperforms NCM. Due to the challenging nature of class incremental learning, some methods employ a buffer of stored samples from previous sessions [24, 39].”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, methodology): “While GDumb [29], which simply used the old training samples to train a new model from scratch at test time, has difficulty to adapt the pre-trained model with limited training samples and thus performs the worst.”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “Existing methods tackling this can be divided into three categories: memory-based, regularization-based, and network-architecture-based [11, 35].”
FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, background): “The results reported here resonate with past works which show that simple methods can be highly effective in CIL [2, 24, 31].”
ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation (2022, background): “The perfect identiﬁer is however often not available in practice [32].”
Task-Free Continual Learning via Online Discrepancy Distance Learning (2022, neutral): “Another kind of approaches for continual learning focuses on the memory system, which usually employs a small memory buffer [1, 5, 6, 28, 36, 44, 59] to store a few past data or trains a generator to provide the replay samples when learning new tasks [38, 43, 46, 47, 52, 58, 53].”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “• Greedy Sampler and Dumb Learner (GDumb) [61]: it trains the model only on buffer datapoints at the task boundaries.”
A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning (2022, methodology): “We also consider popular continual learning methods including regularization-based methods (LwF[13] , EWC [10] ) and rehearsal-based methods (ER[4], GDumb[19], BiC [29], DER++ [1] and Co2L [3]).”
Exploring Example Influence in Continual Learning (2022, neutral): “Rehearsal-based methods [29, 7, 22, 6, 1, 2, 28, 3, 25] tackle the challenge of SP dilemma by retaining a subset of old tasks in a stored memory buffer with bounded resources.”
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, background): “Rehearsal-based approaches [5, 26, 34, 36] replay examples from previous tasks either by storing samples to an external memory buffer or learning a generative model.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “They include non-prompting methods EWC [32], LwF [40] ER [8], GDumb [53], BiC [67], DER++ [4] and Co2L [6], as well as the prompting methods DyTox [14] and L2P [66] and self-supervised learning method CaSSLe [18].”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “(13) GDumb (Prabhu, Torr, and Dokania 2020) is a replay method using a greedy sampler to update the memory.”
Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones (2022, background): “This point have been also raised by GDumb [22] where the authors questioned the progresses by providing a very simple baseline.”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, background): “When the buffer size is 5000 (10% of the CIFAR-100 training set and >20% of the ImageNet-R training set), all rehearsal-based methods are fairly close to GDumb, indicating that performing rehearsal-based continual learning likely provide no performance gain than supervised training on the buffered d”
Constrained Few-shot Class-incremental Learning (2022, methodology): “To address the catastrophic forgetting issue in CIL, rehearsal methods store some old training data, and replay them while learning novel classes during the incremental stage [1, 3, 31, 32, 36, 39].”