PLOP: Learning without Forgetting for Continual Semantic Segmentation

By Arthur Douillard, Yifu Chen, Arnaud Dapogny, Matthieu Cord. 2020

319 citations recorded by the source corpus.

Deep learning approaches are nowadays ubiquitously used to tackle computer vision tasks such as semantic segmentation, requiring large datasets and substantial computational power. Continual learning for semantic segmentation (CSS) is an emerging trend that consists in updating an old model by sequentially adding new classes. However, continual learning methods are usually prone to catastrophic forgetting. This issue is further aggravated in CSS where, at each step, old classes from previous iterations are collapsed into the background. In this paper, we propose Local POD, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships at feature level. Furthermore, we design an entropy-based pseudo-labelling of the background w.r.t. classes predicted by the old model to deal with background shift and avoid catastrophic forgetting of the old classes. Our approach, called PLOP, significantly outperforms state-of-the-art methods in existing CSS scenarios, as well as in newly proposed challenging benchmarks 1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “PLOP [292], RECALL [293], SSUL [294], Self-Training [295] and WILSON [296] explicitly apply the old model to generate pseudo-labels of the old classes.”
ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation (2022, methodology): “SDR [30] and PLOP [9] propose to further regularize a current model in a latent feature space.”
General Incremental Learning with Domain-aware Categorical Representations (2022, methodology): “Furthermore, it is meaningful to apply our method to other vision tasks like video classification [18] and semantic segmentation [8] as future work.”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “(a) Image (b) ILT [19] (c) PLOP [29] (d) PLOP+UCD (e) MiB [23] (f) MiB+UCD (g) GT”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “column spatial KD Channel-wise KD (a) PLOP [27] (b) Ours”
Incremental Learning in Semantic Segmentation from Image Labels (2021, methodology): “A better solution is to incrementally add new classes to the pre-existing model, as done in some recent works [8, 21, 43, 45, 46].”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, neutral): “A different TAB per task leads to better results (56% v.s. 52% in “Last” accuracy) because the network can be more diverse with each TAB specialized to its associated task.”
RECALL: Replay-based Continual Learning in Semantic Segmentation (2021, background): “Recently, increasing attention has been devoted to class-incremental semantic segmentation [29, 5, 30, 11] to learn new categories from new data.”
SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning (2021, methodology): “[3] ﬁrst addressed the semantic drift of BG label, and [7] proposed to use pseudo-labels to augment the BG label, all without exemplar-memory.”
MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, background): “Another related work is PLOP [80] and the aim of PLOP is to avoid overﬁtting on new classes while balancing old classes, which is similar to our Fig.”
Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation (2022, unknown): “Modeling-the-Background (MiB) [4] first clarifies the background shift in CISS, and proposed a distillation-based framework as in CIL.”
Prototype-based Incremental Few-Shot Segmentation (2020, background): “This problem has been extensively studied in image classiﬁcation [4, 9, 18, 21, 22, 38, 53] and recently in segmentation [3, 10, 28, 29, 30, 33, 43].”
ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (2024, methodology): “Moreover, there is a distinct issue in continual segmentation, called semantic drift , as discussed in prior works [2, 5, 11].”
Incremental Nuclei Segmentation from Histopathological Images via Future-class Awareness and Compatibility-inspired Distillation (2024, methodology): “Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. in clinical scenarios, most studies turn to employ distillation techniques [4, 13, 23, 25, 26, 30, 36] to constrain the model updating.”
CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation (2023, methodology): “Most recent works [4, 5, 12] have strug-a gled to alleviate this problem, and the freeze strategy [5] ( i.e ., freezing most of the parameters during all the incremental steps after learning base classes) was shown to be the most efficient way in the state-of-the-art CISS methods [5, 49].”
Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation (2021, background): “This setup, referred here as Continual Semantic Segmentation (CSS), has emerged very recently for specialized applications [11], [12], [13] before being proposed for general segmentation datasets [14], [15], [16].”
Understanding Continual Learning Settings with Data Distribution Drift Analysis (2021, background): “Indeed, in a wide part of continual learning literature, incremental learning is associated with the “disjoint class scenarios” ( virtual concept drifts ) (Belouadah and Popescu, 2018; Wu et al., 2019; Douillard et al., 2020b).”
Endpoints Weight Fusion for Class Incremental Semantic Segmentation (2023, methodology): “In the scenario of CISS, not only the previous data is not accessible due to privacy issues or data storage limitations, but regions of old classes in the newly added dataset are labeled as background, which further exacerbates the model over-fitting.”