Continual Semantic Segmentation with Automatic Memory Sample Selection

By Lanyun Zhu, Tianrun Chen, Jianxiong Yin, Simon See, J. Liu. 2023

81 citations recorded by the source corpus.

Continual Semantic Segmentation (CSS) extends static semantic segmentation by incrementally introducing new classes for training. To alleviate the catastrophic forgetting issue in CSS, a memory buffer that stores a small number of samples from the previous classes is constructed for replay. However, existing methods select the memory samples either randomly or based on a single-factor-driven handcrafted strategy, which has no guarantee to be optimal. In this work, we propose a novel memory sample selection mechanism that selects informative samples for effective replay in a fully automatic way by considering comprehensive factors including sample diversity and class performance. Our mechanism regards the selection operation as a decision-making process and learns an optimal selection policy that directly maximizes the validation performance on a reward set. To facilitate the selection decision, we design a novel state representation and a dual-stage action space. Our extensive experiments on Pascal-VOC 2012 and ADE 20K datasets demonstrate the effectiveness of our approach with state-of-the-art (SOTA) performance achieved, outperforming the second-place one by 12.54% for the 6-stage setting on Pascal-VOC 2012.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Qiwen Wang
5. Ziwei Liu
6. Zhiyuan Qi
7. Jun Liu
8. Hossein Rahmani

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Jun Liu
5. Qiwen Wang
6. Hossein Rahmani
7. Ziwei Liu
8. Zhiyuan Qi

Citations in the corpus

• Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “They used selective memory retrieval, which is shown to be more effective compared to random retrieval in continual analogical reasoning.”
• Class-Incremental Learning: A Survey (2023, background): “SSUL [277] and EM [278] explore the application of exemplar replay in CSS. AMSS [279] proposes a memory sample selection mechanism that selects informative samples for effective replay in a fully automatic way.”
• Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, unknown): “Recently, RL-Replay [65] set a strong replay-based baseline via re-inforcement learning.”
• SAM-Adapter: Adapting Segment Anything in Underperformed Scenes (2023, background): “…the remarkable advancements in deep-learning-based methods such as fully convolutional networks (FCN) [35], encoder-decoder structures [41, 50, 14, 1, 29], dilated convolutions [3, 4, 5, 6, 34], pyramid structures [60, 5, 52, 6, 57], attention modules [51, 16, 62], and transformers [54, 48, 45, 8].”
• Incremental Nuclei Segmentation from Histopathological Images via Future-class Awareness and Compatibility-inspired Distillation (2024, methodology): “In scenarios where previous data can be reused, some studies [22, 45] select samples to help the model review and reinforce knowledge, greatly alleviating catastrophic forgetting.”