ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation

By Youngmin Oh, Donghyeon Baek, Bumsub Ham. 2022

50 citations recorded by the source corpus.

We address the problem of incremental semantic segmentation (ISS) recognizing novel object/stuff categories continually without forgetting previous ones that have been learned. The catastrophic forgetting problem is particularly severe in ISS, since pixel-level ground-truth labels are available only for the novel categories at training time. To address the problem, regularization-based methods exploit probability calibration techniques to learn semantic information from unlabeled pixels. While such techniques are effective, there is still a lack of theoretical understanding of them. Replay-based methods propose to memorize a small set of images for previous categories. They achieve state-of-the-art performance at the cost of large memory footprint. We propose in this paper a novel ISS method, dubbed ALIFE, that provides a better compromise between accuracy and efficiency. To this end, we first show an in-depth analysis on the calibration techniques to better understand the effects on ISS. Based on this, we then introduce an adaptive logit regularizer (ALI) that enables our model to better learn new categories, while retaining knowledge for previous ones. We also present a feature replay scheme that memorizes features, instead of images directly, in order to reduce memory requirements significantly. Since a feature extractor is changed continually, memorized features should also be updated at every incremental stage. To handle this, we introduce category-specific rotation matrices updating the features for each category separately. We demonstrate the effectiveness of our approach with extensive experiments on standard ISS benchmarks, and show that our method achieves a better trade-off in terms of accuracy and efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Han-Jia Ye
  2. De-chuan Zhan
  3. Da-Wei Zhou
  4. Hang Su
  5. Qiwen Wang
  6. Jun Zhu
  7. Ziwei Liu
  8. Liyuan Wang

In Computer Vision and Pattern Recognition

  1. Han-Jia Ye
  2. De-chuan Zhan
  3. Jun Zhu
  4. Da-Wei Zhou
  5. Hang Su
  6. Liyuan Wang
  7. Ziwei Liu
  8. Qiwen Wang

In Radiology, Nuclear Medicine and Imaging

  1. Da-Wei Zhou
  2. De-chuan Zhan
  3. Han-Jia Ye
  4. Liyuan Wang
  5. Jun Zhu
  6. Hang Su
  7. Ziwei Liu
  8. Xingxing Zhang

In Meta & Continual Learning

  1. Han-Jia Ye
  2. De-chuan Zhan
  3. Da-Wei Zhou
  4. Xingxing Zhang
  5. Liyuan Wang
  6. Hang Su
  7. Qiwen Wang
  8. Jun Zhu

Citations in the corpus