Lifelong Learning via Progressive Distillation and Retrospection

By Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, Dahua Lin. 2018

218 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “To further enhance learning plasticity, D+R [154] performs KD from an additional model dedicated to each new task.”
Overcoming Forgetting in Federated Learning on Non-IID Data (2019, background): “One such example is data distillation, in which representative data samples are shared between tasks [5, 6].”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, neutral): “, penalize changes to model predictions) [1,8,22,34,38].”
Continual Learning through Retrieval and Imagination (2022, background): “(2) trying to consolidate the knowledge in the original network by replaying a subset of past examples stored in a memory buffer (Buzzega et al. 2020; Hou et al. 2018; Rannen et al. 2017; Rebuffi et al. 2017; Buzzega et al. 2021).”
Class-Incremental Learning with Strong Pre-trained Models (2022, background): “For the first group, the general approach for incremental training is to impose constraints on parameter changes [3, 13, 32], gradients [17], features [8,11,25], or activations [2,6,10,15,23] when finetuning the network with novel samples.”
Online Continual Learning Via Candidates Voting (2021, methodology): “Representative methods include freezing part of layers [21, 23] and using distillation loss or its variants [26, 13, 6, 35, 18, 19, 25, 14].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “A few other approaches [184, 185] have combined the knowledge distillation andmemory basedmethods.”
Learning with Selective Forgetting (2021, methodology): “This approach can be grouped into data-driven-based [ Li and Hoiem, 2017; Hou et al. , 2018; Dhar et al. , 2019] and weight-constrain-based [Kirkpatrick et al. , 2017; Zenke et al. , 2017; Aljundi et al. , 2018; Chaudhry et al. , 2018a; Lee et al. , 2017; Yu et al. , 2020].”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “Knowledge Distillation in Class-Incremental Learning : Based on the original work proposing knowledge distillation from a large model to a smaller model [22], methods such as Learning Without Forgetting [36], Distillation and Retrospection [23], End-To-End Incremental Learning [7], Global Distillati”
Continual Learning for Neural Machine Translation (2021, methodology): “Most of existing works focus on computer vision tasks (Zenke et al., 2017; Aljundi et al., 2017; Triki et al., 2017; Hou et al., 2018; Aljundi et al., 2018; Hou et al., 2019; Wu et al., 2019).”
Class-Incremental Instance Segmentation via Multi-Teacher Networks (2021, methodology): “The concept of multi-teacher networks is inspired by the human learning process (Hou et al. 2018).”
Split-and-Bridge: Adaptable Class Incremental Learning within a Single Neural Network (2021, background): “Another type of parameter isolation is to temporarily learn a new task in an extra network and then combine it into the previously learned model, which covers P&C (Schwarz et al. 2018), DR (Hou et al. 2018), DMC (Zhang et al. 2020), and GD (Lee et al. 2019).”
Layerwise Optimization by Gradient Decomposition for Continual Learning (2021, neutral): “Example methods of this framework were iCarl [38], End2End [5] and DR [17].”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…al., 2017; Lopez-Paz & Ranzato, 2017; Castro, Marín-Jiménez, Guil, Schmid, & Ala-hari, 2018; Chaudhry et al., 2018; Nguyen, Li, Bui, & Turner, 2018; Hou et al., 2018, 2019; Hayes et al., 2019; Wu, Chen et al., 2019; Lee, Lee, Shin, & Lee, 2019; Belouadah & Popescu, 2019, 2020; Chaudhry et al.,…”
Few-Shot Lifelong Learning (2021, methodology): “To achieve this, many methods (Rebuffi et al. 2017; Saihui et al. 2018; Hou et al. 2019; Castro et al. 2018) use a knowledge distillation loss (Hinton, Vinyals, and Dean 2015).”
Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, methodology): “In [13] authors also use the distillation loss in task incremental learning setting.”
Few-Shot Class-Incremental Learning (2020, methodology): “To mitigate forgetting, most CIL works [32, 2, 35, 13, 48] use the knowledge distillation [12] technique that maintains the network’s output logits corresponding to old classes.”
Knowledge Distillation for Incremental Learning in Semantic Segmentation (2019, background): “e incrementally learning new classes. In [27] a fraction of previous classes is kept as a way to alleviate intransigence of a model, i.e., the inability of a model to update its knowledge. Hou et al. [28] tried to balance between preservation and adaptation of the model via distillation and retrospe”
Incremental Learning Techniques for Semantic Segmentation (2019, background): “Some studies keep a small portion of data belonging to previous tasks and use them to preserve the accuracy on old tasks when dealing with new problems [5, 7, 15, 20, 26, 32].”