Knowledge Distillation for Incremental Learning in Semantic Segmentation

By Umberto Michieli, Pietro Zanuttigh. 2019

109 citations recorded by the source corpus.

Although deep learning architectures have shown remarkable results in scene understanding problems, they exhibit a critical drop of overall performance due to catastrophic forgetting when they are required to incrementally learn to recognize new classes without forgetting the old ones. This phenomenon impacts on the deployment of artificial intelligence in real world scenarios where systems need to learn new and different representations over time. Current approaches for incremental learning deal only with the image classification and object detection tasks. In this work we formally introduce the incremental learning problem for semantic segmentation. To avoid catastrophic forgetting we propose to distill the knowledge of the previous model to retain the information about previously learned classes, whilst updating the current model to learn the new ones. We developed three main methodologies of knowledge distillation working on both the output layers and the internal feature representations. Furthermore, differently from other recent frameworks, we do not store any image belonging to the previous training stages while only the last model is used to preserve high accuracy on previously learned classes. Extensive results were conducted on the Pascal VOC2012 dataset and show the effectiveness of the proposed approaches in different incremental learning scenarios.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “full annotations of both the old and new classes in continual learning [301], [303].”
Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation (2022, background): “Recently, incremental learning has been extensively studied for image segmentation task [13, 33, 45].”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “[19], [50] introduced different distillationbased techniques working both on the output logits and on intermediate features.”
Incremental Learning in Semantic Segmentation from Image Labels (2021, background): “Even if in image classification it has been exhaustively studied [1, 11, 20, 22, 24, 30, 38, 41, 54, 57, 68], in semantic segmentation it is still in its early stages [8,9,21,33,43,45–47].”
RECALL: Replay-based Continual Learning in Semantic Segmentation (2021, methodology): “[8] Umberto Michieli and Pietro Zanuttigh.”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, methodology): “Knowledge Distillation is employed to constraint the decoder to preserve previous knowledge at the output-level and it is implemented as a standard cross-entropy on the output softmax probabilities between old model and current model predictions [5, 6, 1, 3] (see Section 4.”
Continual Learning for Robotics (2019, methodology): “This and related methods have been used in various approaches [171, 144, 48, 136, 68, 160, 33, 105].”
Class-Incremental Learning for Semantic Segmentation Re-Using Neither Old Data Nor Old Labels (2020, methodology): “Previous approaches in semantic segmentation either employ a binary classification loss between each class and the background [14], thereby assuming non-overlapping classes when extending the model, or they make use of labels for old and new classes alike [15], [31].”
Few-Shot Incremental Learning for Label-to-Image Translation (2022, methodology): “Regularization-based methods [1, 5, 20, 54] constrain the changes of models by penalizing the drifting of important parameters or using distillation losses [9, 24, 32, 43].”
Prototype-based Incremental Few-Shot Segmentation (2020, methodology): “This problem has been extensively studied in image classiﬁcation [4, 9, 18, 21, 22, 38, 53] and recently in segmentation [3, 10, 28, 29, 30, 33, 43].”
Online Continual Learning For Interactive Instruction Following Agents (2024, background): “To address these issues, (Buzzega et al., 2020; Michieli & Zanuttigh, 2021; Boschini et al., 2022a) propose using logits instead of storing previous models, saving memory and inference overhead.”
Class Similarity Weighted Knowledge Distillation for Continual Semantic Segmentation (2022, background): “Recent work [9, 25] distill the features of a previous model to reduce this feature divergence problem.”