Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

By Umberto Michieli, Pietro Zanuttigh. 2021

176 citations recorded by the source corpus.

Deep neural networks suffer from the major limitation of catastrophic forgetting old tasks when learning new ones. In this paper we focus on class incremental continual learning in semantic segmentation, where new categories are made available over time while previous training data is not retained. The proposed continual learning scheme shapes the latent space to reduce forgetting whilst improving the recognition of novel classes. Our framework is driven by three novel components which we also combine on top of existing techniques effortlessly. First, prototypes matching enforces latent space consistency on old classes, constraining the encoder to produce similar latent representation for previously seen classes in the subsequent steps. Second, features sparsification allows to make room in the latent space to accommodate novel classes. Finally, contrastive learning is employed to cluster features according to their semantics while tearing apart those of different classes. Extensive evaluation on the Pascal VOC2012 and ADE20K datasets demonstrates the effectiveness of our approach, significantly outperforming state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Hang Su
5. Jun Zhu
6. Liyuan Wang
7. Arthur Douillard
8. Ziwei Liu

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Jun Zhu
4. Ziwei Liu
5. Matthieu Cord
6. Hang Su
7. Da-Wei Zhou
8. Arthur Douillard

In Meta & Continual Learning

1. Arthur Douillard
2. Han-Jia Ye
3. De-chuan Zhan
4. Liyuan Wang
5. Da-Wei Zhou
6. Xingxing Zhang
7. Hang Su
8. Jun Zhu

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “SDR [291] introduces contrastive learning into distillation of latent representations, where pixels of the same class are clustered and pixels of different classes are separated.”
• ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation (2022, methodology): “Specifically, SDR minimizes distances between features of the same category [6], while PLOP distills intermediate features using a multi-scale strip pooling technique [8, 17].”
• Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “There is two main difference between our method and SDR: (i) we compare feature vectors between themselves in the feature space, not with prototypes, (ii) we use an uncertainty estimation mechanism to guide contrastive learning.”
• Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “Michieli and Zanuttigh [65] proposed several strategies in the latent space complementary to knowledge distillation including prototype matching, contrastive learning, and feature sparsity.”
• RECALL: Replay-based Continual Learning in Semantic Segmentation (2021, methodology): “[7] Umberto Michieli and Pietro Zanuttigh.”
• Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation (2021, unknown): “While the problem of incremental learning has been traditionally addressed in object recognition [6, 22, 21] and detection [40], much less attention has been devoted to semantic segmentation [27, 28, 5].”
• Continual Learning of Semantic Segmentation using Complementary 2D-3D Data Representations (2021, background): “A limited number of works has explored semantic segmentation in continual learning [20, 30, 6, 10, 9, 19].”
• Incremental Nuclei Segmentation from Histopathological Images via Future-class Awareness and Compatibility-inspired Distillation (2024, background): “Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. in clinical scenarios, most studies turn to employ distillation techniques [4, 13, 23, 25, 26, 30, 36] to constrain the model updating.”
• Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, unknown): “While the research focus of continual learning is on classification [1, 6, 11, 22, 25, 38, 41, 50, 58], continual segmentation started to thrive recently [3, 5, 7, 12, 32, 36, 40, 51, 57, 60, 61, 65].”
• CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation (2023, methodology): “Besides, we provide the results of the prior state-of-the-art CISS methods, including MiB [4], SDR [26], PLOP [12], SSUL [5], RCIL [45], and MicroSeg [49].”
• Weakly-Supervised Continual Learning for Class-Incremental Segmentation (2022, methodology): “Hence, we draw inspiration from [11, 12] which design customized regularization to address the two identified pitfalls.”
• Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation (2021, methodology): “both the proposed PLOP and PLOPLong outperform stateof-the-art approaches, MiB [15], SDR [76], and GIFS [75] on all evaluated settings by a significant margin.”