Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation

By Guanglei Yang, Enrico Fini, Dan Xu, Paolo Rota, Ming Ding, Moin Nabi, Xavier Alameda-Pineda, Elisa Ricci. 2022

97 citations recorded by the source corpus.

A fundamental and challenging problem in deep learning is catastrophic forgetting, the tendency of neural networks to fail to preserve the knowledge acquired from old tasks when learning new tasks. This problem has been widely investigated in the research community and several Incremental Learning approaches have been proposed in the past years. While earlier works in computer vision have mostly focused on image classification and object detection, more recently some IL approaches for semantic segmentation have been introduced. These previous works showed that, despite its simplicity, knowledge distillation can be effectively employed to alleviate catastrophic forgetting. In this paper, we follow this research direction and, inspired by recent literature on contrastive learning, we propose a novel distillation framework, Uncertainty-aware Contrastive Distillation. In a nutshell, is operated by introducing a novel distillation loss that takes into account all the images in a mini-batch, enforcing similarity between features associated to all the pixels from the same classes, and pulling apart those corresponding to pixels from different classes. Our experimental results demonstrate the advantage of the proposed distillation technique, which can be used in synergy with previous IL approaches, and leads to state-of-art performance on three commonly adopted benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Hang Su
5. Qiwen Wang
6. Jun Zhu
7. Ziwei Liu
8. Liyuan Wang

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Jun Zhu
4. Da-Wei Zhou
5. Hang Su
6. Ziwei Liu
7. Yunchao Wei
8. Liyuan Wang

In Representation & Self-Supervised Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Liyuan Wang
5. Chi Harold Liu
6. Jun Zhu
7. Hang Su
8. Beomyoung Kim

In Meta & Continual Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Xingxing Zhang
5. Liyuan Wang
6. Hang Su
7. Qiwen Wang
8. Jun Zhu

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “SDR [302] and UCD [470] introduce contrastive learning into distillation of latent representations, where pixels of the same class are clustered and pixels of different classes are separated.”
• Deep Class-Incremental Learning: A Survey (2023, background): “, semantic segmentation [164], [165], [166], [167], [174], object detection [175], [176], [177], [178], [179], [180], [181], video understanding [95], [133], medical surgery [182], [183], language model [184], [185], [186], [187], language-vision model [188], [189], etc.”
• Class-Incremental Learning: A Survey (2023, methodology): “Furthermore, it has shown the potential to apply the techniques in classification into other tasks, e.g. , semantic segmentation [193], [238], [239], [240], [241], object detection [137], [242], [243], [244], [245], [246], [247], video understanding [91], [161], medical surgery [248], [249],…”
• ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (2024, methodology): “Recently, various continual segmentation methods [2, 5, 11, 35, 37, 40, 46, 47, 49] have emerged, addressing the key challenges and showing notable improvements.”
• Saving 100x Storage: Prototype Replay for Reconstructing Training Sample Distribution in Class-Incremental Semantic Segmentation (2023, background): “Among them, MiB [2], SDR [23], PLOP [9], SSUL [3], RCIL [31], RBC [33], DKD [1], UCD [29], and AWT [12] are non-replay-based, while RECALL [20], SSUL-M [3], and DKD-M [1] are replay-based methods.”
• CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation (2023, background): “An intuitive idea for discriminative representation is to compute prototypes (category centroids) for each category and apply contrastive learning to improve the diversity among categories [26, 44].”