SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning

By Sungmin Cha, Beomyoung Kim, Y. Yoo, Taesup Moon. 2021

135 citations recorded by the source corpus.

This paper introduces a solid state-of-the-art baseline for a class-incremental semantic segmentation (CISS) problem. While the recent CISS algorithms utilize variants of the knowledge distillation (KD) technique to tackle the problem, they failed to fully address the critical challenges in CISS causing the catastrophic forgetting; the semantic drift of the background class and the multi-label prediction issue. To better address these challenges, we propose a new method, dubbed SSUL-M (Semantic Segmentation with Unknown Label with Memory), by carefully combining techniques tailored for semantic segmentation. Specifically, we claim three main contributions. (1) defining unknown classes within the background class to help to learn future classes (help plasticity), (2) freezing backbone network and past classifiers with binary cross-entropy loss and pseudo-labeling to overcome catastrophic forgetting (help stability), and (3) utilizing tiny exemplar memory for the first time in CISS to improve both plasticity and stability. The extensively conducted experiments show the effectiveness of our method, achieving significantly better performance than the recent state-of-the-art baselines on the standard benchmark datasets. Furthermore, we justify our contributions with thorough ablation analyses and discuss different natures of the CISS problem compared to the traditional class-incremental learning targeting classification. The official code is available at https://github.com/clovaai/SSUL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Hang Su
5. Qiwen Wang
6. Jun Zhu
7. Ziwei Liu
8. Liyuan Wang

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Jun Zhu
4. Da-Wei Zhou
5. Hang Su
6. Ziwei Liu
7. Yunchao Wei
8. Liyuan Wang

In Radiology, Nuclear Medicine and Imaging

1. Da-Wei Zhou
2. De-chuan Zhan
3. Han-Jia Ye
4. Liyuan Wang
5. Jun Zhu
6. Hang Su
7. Ziwei Liu
8. Xingxing Zhang

In Meta & Continual Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Xingxing Zhang
5. Liyuan Wang
6. Hang Su
7. Qiwen Wang
8. Jun Zhu

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “PLOP [292], RECALL [293], SSUL [294], Self-Training [295] and WILSON [296] explicitly apply the old model to generate pseudo-labels of the old classes.”
• ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation (2022, methodology): “Following [2, 3, 9, 26, 30], we adopt DeepLab-V3 [5] with ResNet-101 [14].”
• Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation (2022, unknown): “Figure 1: (a) Comparison of our proposed MicroSeg and the state-of-the-art SSUL [5].”
• ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (2024, methodology): “Moreover, there is a distinct issue in continual segmentation, called semantic drift , as discussed in prior works [2, 5, 11].”
• Deep Class-Incremental Learning: A Survey (2023, unknown): “Replay is a simple yet effective strategy, which has been widely applied to image-based camera localization [129], semantic segmentation [130], [131], [132], video classification [133], and action recognition [95].”
• Saving 100x Storage: Prototype Replay for Reconstructing Training Sample Distribution in Class-Incremental Semantic Segmentation (2023, methodology): “Note that although previous replay-based CISS works [1, 3, 20] consider the main function of replay to be combating forgetting, we argue that its more crucial role is in learning new classes, as confirmed by [1, 3], where the versions with replay improve the performance on new classes more than old…”
• Incremental Nuclei Segmentation from Histopathological Images via Future-class Awareness and Compatibility-inspired Distillation (2024, methodology): “In order to improve both stability and plasticity, a few studies [5, 39] propose to learn future classes in advance to prepare for learning new classes.”
• Continual Semantic Segmentation with Automatic Memory Sample Selection (2023, methodology): “Following [7], the memory length |M| is 100 and 300 for Pascal-VOC 2012 and ADE20K, respectively.”
• Class-Incremental Learning: A Survey (2023, background): “SSUL [277] and EM [278] explore the application of exemplar replay in CSS. AMSS [279] proposes a memory sample selection mechanism that selects informative samples for effective replay in a fully automatic way.”
• Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, unknown): “While the research focus of continual learning is on classification [1, 6, 11, 22, 25, 38, 41, 50, 58], continual segmentation started to thrive recently [3, 5, 7, 12, 32, 36, 40, 51, 57, 60, 61, 65].”
• CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation (2023, methodology): “Previous studies [5, 49] have highlighted the advantages of the freeze strategy in CISS.”