Gradient Regularized Contrastive Learning for Continual Domain Adaptation
By Peng Su, Shixiang Tang, Peng Gao, Di Qiu, Ni Zhao, Xiaogang Wang. 2020
72 citations recorded by the source corpus.
Human beings can quickly adapt to environmental changes by leveraging learning experience. However, adapting deep neural networks to dynamic environments by machine learning algorithms remains a challenge. To better understand this issue, we study the problem of continual domain adaptation, where the model is presented with a labelled source domain and a sequence of unlabelled target domains. The obstacles in this problem are both domain shift and catastrophic forgetting. We propose Gradient Regularized Contrastive Learning (GRCL) to solve the obstacles. At the core of our method, gradient regularization plays two key roles: (1) enforcing the gradient not to harm the discriminative ability of source features which can, in turn, benefit the adaptation ability of the model to target domains; (2) constraining the gradient not to increase the classification loss on old target domains, which enables the model to preserve the performance on old target domains when adapting to an in-coming target domain. Experiments on Digits, DomainNet and Office-Caltech benchmarks demonstrate the strong performance of our approach when compared to the state-of-the-art.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Joost van de Weijer
- Mingkui Tan
- Luis Herranz
- Shipeng Yan
- Massimiliano Mancini
- Chuang Gan
- Shuaicheng Niu
- Yifan Zhang
Citations in the corpus
- Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation (2022, background): “Recently, GRCL [43] regularizes the gradient of losses to learn discriminative features and preserve the previous knowledge, respectively.”
- General Incremental Learning with Domain-aware Categorical Representations (2022, background): “Problem Settings Most previous works focus on either class incremental learning [4, 6, 9, 13, 23, 28, 30, 35, 37, 38] or domain incremental learning [29, 30, 33].”
- Layerwise Optimization by Gradient Decomposition for Continual Learning (2021, neutral): “This phenomenon, known as catastrophic forgetting , invokes more effective algorithms for continual learning (CL), the goal of which is to learn consecutive tasks without severe performance degradation on previous tasks [5, 30, 34, 38, 44, 43, 57, 57, 50].”
- Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “Gradient regularized contrastive learning [217] leverages the contrastive loss to push unlabeled target samples towards the most similar labeled source ones.”
- Generalized Source-free Domain Adaptation (2021, methodology): “Recently, a few works [4, 26, 36] have emerged that aim to tackle the Continual Domain Adaptation (CDA) problem.”