Representation Compensation Networks for Continual Semantic Segmentation

By Chang-Bin Zhang, Jianqiang Xiao, Xialei Liu, Ying-Cong Chen, Mingg-Ming Cheng. 2022

143 citations recorded by the source corpus.

In this work, we study the continual semantic segmentation problem, where the deep neural networks are required to incorporate new classes continually without catastrophic forgetting. We propose to use a structural re-parameterization mechanism, named representation compensation (RC) module, to decouple the representation learning of both old and new knowledge. The RC module consists of two dynamically evolved branches with one frozen and one trainable. Besides, we design a pooled cube knowledge distillation strategy on both spatial and channel dimensions to further enhance the plasticity and stability of the model. We conduct experiments on two challenging continual semantic segmentation scenarios, continual class segmentation and continual domain segmentation. Without any extra computational overhead and parameters during inference, our method outperforms state-of-the-art performance. The code is available at https://github.com/zhangchbin/RCIL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Hang Su
5. Qiwen Wang
6. Jun Zhu
7. Ziwei Liu
8. Liyuan Wang

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Jun Zhu
4. Ziwei Liu
5. Da-Wei Zhou
6. Joost van de Weijer
7. Hang Su
8. Yunchao Wei

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “RCIL [290] reparameterizes the network into two parallel branches, where the old branch is frozen for KD between intermediate layers.”
• Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, unknown): “While the research focus of continual learning is on classification [1, 6, 11, 22, 25, 38, 41, 50, 58], continual segmentation started to thrive recently [3, 5, 7, 12, 32, 36, 40, 51, 57, 60, 61, 65].”
• ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (2024, methodology): “Recently, various continual segmentation methods [2, 5, 11, 35, 37, 40, 46, 47, 49] have emerged, addressing the key challenges and showing notable improvements.”
• Deep Class-Incremental Learning: A Survey (2023, background): “, semantic segmentation [164], [165], [166], [167], [174], object detection [175], [176], [177], [178], [179], [180], [181], video understanding [95], [133], medical surgery [182], [183], language model [184], [185], [186], [187], language-vision model [188], [189], etc.”
• CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation (2023, methodology): “Besides, we provide the results of the prior state-of-the-art CISS methods, including MiB [4], SDR [26], PLOP [12], SSUL [5], RCIL [45], and MicroSeg [49].”
• Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection (2023, background): “While most studies in incremental learning are based on image classification [2, 31, 34, 50], recently it has been studied in the context of object detection [7, 9, 22, 48, 55] and semantic segmentation [13, 20, 64].”
• Saving 100x Storage: Prototype Replay for Reconstructing Training Sample Distribution in Class-Incremental Semantic Segmentation (2023, background): “Compared to distillations that operate on the entire spatial region [1, 9, 31], our OCFM loss specifically targets old-class regions, preserving these features to collaborate with our replay strategy and combat catastrophic forgetting.”
• CoMFormer: Continual Learning in Semantic and Panoptic Segmentation (2022, methodology): “RCIL was not re-implemented since it is not possible to extend their approach to non-convolutional architectures like ours.”
• Incremental Nuclei Segmentation from Histopathological Images via Future-class Awareness and Compatibility-inspired Distillation (2024, background): “Architecture-based approaches [16, 38, 44] dynamically grow the network to extend model capacity for new tasks.”
• Class-Incremental Learning: A Survey (2023, methodology): “Additionally, RCIL [240] also explores the application of dynamic networks in CSS.”