Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild

By Kibok Lee, Kimin Lee, Jinwoo Shin, Honglak Lee. 2019

237 citations recorded by the source corpus.

Lifelong learning with deep neural networks is well-known to suffer from catastrophic forgetting: the performance on previous tasks drastically degrades when learning a new task. To alleviate this effect, we propose to leverage a large stream of unlabeled data easily obtainable in the wild. In particular, we design a novel class-incremental learning scheme with (a) a new distillation loss, termed global distillation, (b) a learning strategy to avoid overfitting to the most recent task, and (c) a confidence-based sampling method to effectively leverage unlabeled external data. Our experimental results on various datasets, including CIFAR and ImageNet, demonstrate the superiority of the proposed methods over prior methods, particularly when a stream of unlabeled data is accessible: our method shows up to 15.8% higher accuracy and 46.5% less forgetting compared to the state-of-the-art method. The code is available at https://github.com/kibok90/iccv2019-inc.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “GD [124] further distills knowledge on the large stream of unlabeled data available in the wild.”
Learnability and Algorithm for Continual Learning (2023, methodology): “, 2018), GD (Lee et al., 2019) without external data, DER++ (Buzzega et al.”
A Theoretical Study on Solving Continual Learning (2022, methodology): “Using regularization [12, 13] to minimize changes to model parameters learned from previous tasks is a popular approach [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26].”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, positive): “SETS is completely different from CL and SSCL in several aspects: (i) our SETS focuses on a large-scale task sequence (more than 72K), while CL and SSCL are not scalable to this large-scale setting; (ii) our goal is to enable the model to generalize to the unseen tasks from all the previous task dis”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “A number of these methods [42, 19, 13, 7, 2, 47, 8, 30] use variants of knowledge distillation [18], where the model and its predictions corresponding to the previous task are utilized to prevent the current task’s model from diverging too much from its previous state.”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “We compare the proposed method against competitive and wellestablished methods(4), including 5 regularization-based methods using importance measure (EWC [17], MAS [1], MUC-MAS [25], SI [58], and CPR [5]), 2 regularization-based methods using knowledge distillation (LwF [23] and GDWILD [20]), 1 arch”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “In computer vision, several researchers used the term incremental learning to mean Class-IL as they learn like any other Class-IL method with multiple classes per task (Wu et al. 2019; Castro et al. 2018; Liu et al. 2020; Lee et al. 2019; Belouadah and Popescu 2019, 2020).”
Online Continual Learning Via Candidates Voting (2021, methodology): “Representative methods include freezing part of layers [21, 23] and using distillation loss or its variants [26, 13, 6, 35, 18, 19, 25, 14].”
Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, neutral): “The representative works include LwF [25], EBLL [32], GD-WILD [23], etc. Rehearsal methods : This line of works preserve existing information by replaying data from previous tasks.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[71] pointed out that increasing number of data instances for training each task can help mitigate catastrophic forgetting.”
Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting (2021, methodology): “Table 4 conﬁrms our ﬁnding that with little amount of data knowledge distillation (KD) [4, 10] performs worse than with”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “, penalize changes to model predictions) [7, 23, 33, 36, 52].”
Continual Learning by Using Information of Each Class Holistically (2021, methodology): “One typical approach is to memorize some exemplars of old tasks to perform knowledge distillation (Wu et al. 2019; Castro et al. 2018; Liu et al. 2020; Lee et al. 2019; Belouadah and Popescu 2019, 2020) to deal with CF.”
Split-and-Bridge: Adaptable Class Incremental Learning within a Single Neural Network (2021, background): “Another type of parameter isolation is to temporarily learn a new task in an extra network and then combine it into the previously learned model, which covers P&C (Schwarz et al. 2018), DR (Hou et al. 2018), DMC (Zhang et al. 2020), and GD (Lee et al. 2019).”
Layerwise Optimization by Gradient Decomposition for Continual Learning (2021, neutral): “We compare our approach with other competing methods, including LWF [29], iCarl [38], DR [17], End2End [5], LUCR 2 [18], GD [26], MUC [11] and TPL [51].”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…Schmid, & Ala-hari, 2018; Chaudhry et al., 2018; Nguyen, Li, Bui, & Turner, 2018; Hou et al., 2018, 2019; Hayes et al., 2019; Wu, Chen et al., 2019; Lee, Lee, Shin, & Lee, 2019; Belouadah & Popescu, 2019, 2020; Chaudhry et al., 2019; Riemer et al., 2019; Aljundi, Belilovsky et al., 2019; Aljundi,…”