Modeling the Background for Incremental Learning in Semantic Segmentation

By Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulò, Elisa Ricci, Barbara Caputo. 2020

373 citations recorded by the source corpus.

Despite their effectiveness in a wide range of tasks, deep architectures suffer from some important limitations. In particular, they are vulnerable to catastrophic forgetting, i.e. they perform poorly when they are required to update their model as new classes are available but the original training set is not retained. This paper addresses this problem in the context of semantic segmentation. Current strategies fail on this task because they do not consider a peculiar aspect of semantic segmentation: since each training step provides annotation only for a subset of all possible classes, pixels of the background class (i.e. pixels that do not belong to any other classes) exhibit a semantic distribution shift. In this work we revisit classical incremental learning methods, proposing a new distillation-based framework which explicitly accounts for this shift. Furthermore, we introduce a novel strategy to initialize classifier's parameters, thus preventing biased predictions toward the background class. We demonstrate the effectiveness of our approach with an extensive evaluation on the Pascal-VOC 2012 and ADE20K datasets, significantly outperforming state of the art incremental learning methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Note that, ak,j can be adapted to other forms depending on the task type, such as average precision (AP) for object detection [384], Intersection-over-Union (IoU) for semantic segmentation [56], Fréchet Inception Distance (FID) for image generation [452], normalized reward for reinforcement learning”
ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation (2022, methodology): “The CCE loss [2] additionally computes a probability for unlabeled regions by summing the probabilities over all previous categories as follows: where p t cce ( p ) = k ∈ C t prev p tk ( p ) .”
Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation (2022, methodology): “The incremental segmentation methods (MiB[3] and EMNet[45]) perform poorly in few-shot settings, since they generally require large number of samples for new-class learning.”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “[23] proposed Modeling the Background for Incremental Learning (MiB), a distillation-based approach to learn the predictor φθk at step k given a training set T k by distilling knowledge using the predictions of model φθk−1 on the previous step.”
Incremental Learning in Semantic Segmentation from Image Labels (2021, methodology): “A better solution is to incrementally add new classes to the pre-existing model, as done in some recent works [8, 21, 43, 45, 46].”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “This increased diversity has a drawback: the memory overhead is too important (97M v.s. 10M parameters).”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[63] pointed out that simple adaptation of ideas from image classification which regards the previously learned tasks as background during the learning process of the new task can even encourage rather than mitigate catastrophic forgetting.”
RECALL: Replay-based Continual Learning in Semantic Segmentation (2021, methodology): “However, using another semantic segmentation dataset raises some concerns about the fact that the pre-training data could contain information about the tasks to be learned in the incremental steps, thus following previous works [6, 7, 1] we decided for a more conservative approach in the main paper ”
An EM Framework for Online Incremental Learning of Semantic Segmentation (2021, positive): “Notably, MiB introduces a more natural setting requiring only the pixel annotations of novel classes, which is also adopted by our work.”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, methodology): “The regularization-based methods [23, 42, 4, 30, 2] mitigate forgetting by imposing a regularization to restrain the model parameters.”
SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning (2021, methodology): “The first challenge, as mentioned in the Introduction and in previous work [3, 7], comes from the semantic drift of the background (BG) label.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…to perform continual semantic segmentation of medical images (Ozdemir, Fuernstahl, & Goksel, 2018; Ozdemir & Goksel, 2019), remote sensing data (Tasar, Tarabalka, & Alliez, 2019; Wu, Wang et al., 2019), and on standard computer vision benchmarks (Cermelli, Mancini, Bulo, Ricci, & Caputo, 2020).”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “, continual object detection [1, 62], continual semantic segmentation [11, 48], or continual learning for robotics applications [23, 37].”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, methodology): “RGB label FT LwF [2] ILT [5] CIL [3] MiB [1] SDR (ours) offline Figure 2.”
Self-Training for Class-Incremental Semantic Segmentation (2020, methodology): “We compare different methods in three different scenarios as in [11]; 19-1 means that we ﬁrst learn a model with ﬁrst 19 classes and then learn the remaining class as the second task.”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, methodology): “[7] are the first to address this background shift problem along with catastrophic forgetting.”
Online Continual Learning under Extreme Memory Constraints (2020, background): “Recently, the concept of distillation has been extended to attention and segmentation maps [8,6].”
MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, methodology): “199) as done in MiB [75] and the results are shown in Table 3.”