Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer Learning

By Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, Jianmin Wang. 2019

194 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, background): “On the other hand, when fine-tuning is performed with insufficient training data, it may forget the pre-trained knowledge that is beneficial to the downstream task, thereby hurting generalization performance [14].”
AFEC: Active Forgetting of Negative Transfer in Continual Learning (2021, neutral): “[3] developed a method to mitigate negative transfer when fine-tuning tasks on a pretrained network.”
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020, background): “Thus, a future direction for potential improvement for intermediate-task training may be integrating the MLM objective into intermediate-task training or bounding network parameter changes to reduce catastrophic forgetting (Kirkpatrick et al., 2016; Chen et al., 2019).”
A Review of Deep Transfer Learning and Recent Advancements (2022, methodology): “The primary issue in this approach is known as catastrophic forgetting dilemma since the obtained knowledge can be partially or totally wiped out [3, 4], making it a foremost necessity for this method to use similar source and target data.”
MeLL: Large-scale Extensible User Intent Classification for Dialogue Systems with Meta Lifelong Learning (2021, methodology): “Typical methods include experience replay [16, 20, 40], knowledge distillation [6, 26], transfer learning [5, 19, 39], etc.”
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020, background): “…et al., 2019; Liu et al., 2019; Lan et al., 2020), previous works explore many finetuning tricks to reduce catastrophic forgetting for adaptation of the deep pretrained LMs (Howard and Ruder, 2018; Sun et al., 2019; Zhang et al., 2019; Chen et al., 2019; Jiang et al., 2019; Lee et al., 2020).”
Feature Transformation Ensemble Model with Batch Spectral Regularization for Cross-Domain Few-Shot Classification (2020, background): “Previous work [2] shows that penalizing smaller singular values of a feature matrix can help mitigate negative transfer in ﬁne-tuning.”
Zoo-Tuning: Adaptive Transfer from a Zoo of Models (2021, background): “However, most of the existing fine-tuning approaches only transfer from a single pretrained deep network to the target task (Xuhong et al., 2018; Li et al., 2019; Chen et al., 2019).”
CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation (2021, unknown): “In addition, to address catastrophic forgetting in class-incremental learning, rehearsal [7] principle as well as KD loss were proposed by Rebuffi et al.”
Overcoming Negative Transfer: A Survey (2020, background): “Chen et al. [85] discovered that features with small singular values have low transferability during deep network ﬁne-tune process.”
Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, methodology): “However, they suffer from the issue of negative transfer [9], since downstream tasks are often different from the pre-training contrastive task.”
Decoupled Mixup for Data-efficient Learning (2022, unknown): “Besides the vanilla Fine-Tuning baseline, we compare current state-of-the-art TL methods, including BSS (Xinyang et al., 2019), Co-Tuning (You et al., 2020), and Self-Tuning (Ximei et al., 2021).”
Deep Learning for Visual Tracking: A Comprehensive Survey (2019, background): “learning penalties [257], batch spectral shrinkage [258], or lifelong learning [259].”
A Survey on Negative Transfer (2020, background): “3) Interpretable Positive Transfer: The interpretability of TL is challenging and rarely studied, especially for neural network based approaches [97].”