Continually Learning Self-Supervised Representations with Projected Functional Regularization
By Alex Gomez-Villa, Bartłomiej Twardowski, Lu Yu, Andrew D. Bagdanov, Joost van de Weijer. 2021
58 citations recorded by the source corpus.
Recent self-supervised learning methods are able to learn high-quality image representations and are closing the gap with supervised approaches. However, these methods are unable to acquire new knowledge incrementally – they are, in fact, mostly used only as a pre-training phase over IID data. In this work we investigate self-supervised methods in continual learning regimes without any replay mechanism. We show that naive functional regularization, also known as feature distillation, leads to lower plasticity and limits continual learning performance. Instead, we propose Projected Functional Regularization in which a separate temporal projection network ensures that the newly learned feature space preserves information of the previous one, while at the same time allowing for the learning of new features. This prevents forgetting while maintaining the plasticity of the learner. Comparison with other incremental learning approaches applied to self-supervision demonstrates that our method obtains competitive performance in different scenarios and on multiple datasets.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yuki M. Asano
- Joost van de Weijer
- Huy Tran
- Soheil Kolouri
- Hamed Pirsiavash
- Mert Kilickaya
- Zihao Wu
- Mohammad Rostami
In Computer Vision and Pattern Recognition
- Joost van de Weijer
- Yuki M. Asano
- Luc Van Gool
- Ming-Hsuan Yang
- Andrew Zisserman
- Cordelia Schmid
- Andrea Vedaldi
- Dacheng Tao
Citations in the corpus
- Towards Label-Efficient Incremental Learning: A Survey (2023, methodology): “Projected Functional Regularization [Gomez-Villa et al., 2022] is a regularization-based technique, very similar to CaSSLe.”
- Is Multi-Task Learning an Upper Bound for Continual Learning? (2022, background): “Recently, there have been several works that focus on continual self-supervised learning [17, 18, 19], which hold promise for the next generation of CL algorithms.”