An EM Framework for Online Incremental Learning of Semantic Segmentation

By Shipeng Yan, Jiale Zhou, Jiangwei Xie, Songyang Zhang, Xuming He. 2021

45 citations recorded by the source corpus.

Incremental learning of semantic segmentation has emerged as a promising strategy for visual scene interpretation in the open-world setting. However, it remains challenging to acquire novel classes in an online fashion for the segmentation task, mainly due to its continuously-evolving semantic label space, partial pixelwise ground-truth annotations, and constrained data availability. To address this, we propose an incremental learning strategy that can fast adapt deep segmentation models without catastrophic forgetting, using a streaming input data with pixel annotations on the novel classes only. To this end, we develop a unified learning strategy based on the Expectation-Maximization (EM) framework, which integrates an iterative relabeling strategy that fills in the missing labels and a rehearsal-based incremental learning step that balances the stability-plasticity of the model. Moreover, our EM algorithm adopts an adaptive sampling method to select informative training data and a class-balancing training strategy in the incremental model updates, both improving the efficacy of model learning. We validate our approach on the PASCAL VOC 2012 and ADE20K datasets, and the results demonstrate its superior performance over the existing incremental methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Qiwen Wang
5. Ziwei Liu
6. Jiashi Feng
7. Zhenguo Li
8. Lanqing Hong

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Jiashi Feng
4. Da-Wei Zhou
5. Ziwei Liu
6. Dacheng Tao
7. Luc Van Gool
8. Trevor Darrell

In Radiology, Nuclear Medicine and Imaging

1. Da-Wei Zhou
2. De-chuan Zhan
3. Han-Jia Ye
4. Yifan Zhang
5. Shipeng Yan
6. Xuming He
7. Dongyan Zhao
8. B. Liu

In Meta & Continual Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Qiwen Wang
5. Ziwei Liu
6. Zhiyuan Qi
7. Lanqing Hong
8. Zhenguo Li

Citations in the corpus

• Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, background): “Existing literature focuses mostly on supervised continual learning [50,53,41], which mainly falls into the following four groups.”
• General Incremental Learning with Domain-aware Categorical Representations (2022, background): “Problem Settings Most previous works focus on either class incremental learning [4, 6, 9, 13, 23, 28, 30, 35, 37, 38] or domain incremental learning [29, 30, 33].”
• How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, methodology): “Replay-based CL methods overcome forgetting by saving samples of previous tasks in a replay buffer (Rebuffi et al., 2017; Rolnick et al., 2019; Wang et al., 2021; Yan et al., 2021b) and using them to regularize the learning of new tasks.”
• Online Continual Learning through Mutual Information Maximization (2022, methodology): “…et al., 2017; de Masson d’Autume et al., 2019; Wu et al., 2019; Hou et al., 2019; Zhao et al., 2021; Korycki & Krawczyk, 2021; Sokar et al., 2021; Yan et al., 2021; Hu et al., 2021; Wang et al., 2022), which uses a memory buffer to store some previous data and replays it in training a new task.”
• Class-Incremental Learning: A Survey (2023, background): “SSUL [277] and EM [278] explore the application of exemplar replay in CSS. AMSS [279] proposes a memory sample selection mechanism that selects informative samples for effective replay in a fully automatic way.”