Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data
By Onur Tasar, Yuliya Tarabalka, Pierre Alliez. 2018
144 citations recorded by the source corpus.
In spite of remarkable success of the convolutional neural networks on semantic segmentation, they suffer from catastrophic forgetting: a significant performance drop for the already learned classes when new classes are added on the data having no annotations for the old classes. We propose an incremental learning methodology, enabling to learn segmenting new classes without hindering dense labeling abilities for the previous classes, although the entire previous data are not accessible. The key points of the proposed approach are adapting the network to learn new as well as old classes on the new training data, and allowing it to remember the previously learned information for the old classes. For adaptation, we keep a frozen copy of the previously trained network, which is used as a memory for the updated network in the absence of annotations for the former classes. The updated network minimizes a loss function, which balances the discrepancy between outputs for the previous classes from the memory and updated networks, and the misclassification rate between outputs for the new classes from the updated network and the new ground-truth. For remembering, we either regularly feed samples from the stored, little fraction of the previous data or use the memory network, depending on whether the new data are collected from completely different geographic areas or from the same city. Our experimental results prove that it is possible to add new classes to the network, while maintaining its performance for the previous classes, despite the whole previous training data are not available.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology, Remote Sensing, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Joost van de Weijer
- Arthur Douillard
- Fabio Cermelli
- Matthieu Cord
- Xialei Liu
- Elisa Ricci
- Bartłomiej Twardowski
- Andrew D. Bagdanov
In Computer Vision and Pattern Recognition
- Joost van de Weijer
- Matthieu Cord
- Arthur Douillard
- Luc Van Gool
- Xialei Liu
- Elisa Ricci
- Fabio Cermelli
- Bernt Schiele
In Media Technology
- Nassim Ammour
- Simone Scardapane
- Valerio Marsocci
- Joost van de Weijer
- Andrew D. Bagdanov
- Xialei Liu
- Bertrand Le Saux
- Elisa Ricci
In Remote Sensing
- Pietro Zanuttigh
- Bertrand Le Saux
- Haifeng Li
- Kun Fu
- Nassim Ammour
- Xian Sun
- Umberto Michieli
- Chao Tao
In Meta & Continual Learning
- Joost van de Weijer
- Arthur Douillard
- Fabio Cermelli
- Xialei Liu
- Matthieu Cord
- Andrew D. Bagdanov
- Bartłomiej Twardowski
- Pietro Zanuttigh
Citations in the corpus
- Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “Similarly, [23] proposes a framework based on two purposes: adapting and remembering.”
- Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “To the best of our knowledge, the problem of incremental learning in semantic segmentation has been addressed only in [19], [20], [22], [47].”
- Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Aside from image classification, a few works have focused on other computer vision problems, such as image semantic segmentation [106,107], object detection [108], and analogical reasoning [109].”
- Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…to perform continual semantic segmentation of medical images (Ozdemir, Fuernstahl, & Goksel, 2018; Ozdemir & Goksel, 2019), remote sensing data (Tasar, Tarabalka, & Alliez, 2019; Wu, Wang et al., 2019), and on standard computer vision benchmarks (Cermelli, Mancini, Bulo, Ricci, & Caputo, 2020).”
- Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, background): “Hence, it emerged the need for continual approaches specifically targeted to solve the semantic segmentation task [47, 58, 42, 43, 33, 4].”
- Self-Training for Class-Incremental Semantic Segmentation (2020, background): “It is also investigated in remote sensing [9] and medical data [47] for incremental semantic segmentation.”
- Modeling the Background for Incremental Learning in Semantic Segmentation (2020, background): “In contrast with previous works, we do not limit our analysis to medical [24] or remote sensing data [30] and we do not impose any restrictions on how the label space should change across different learning steps [22].”
- Knowledge Distillation for Incremental Learning in Semantic Segmentation (2019, background): “Another example of this family is the first work on incremental learning for semantic segmentation [29], which however focused on a very specific setting related to satellite images and has several limitations when applied to generic semantic segmentation problems.”
- Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, methodology): “Incremental semantic segmentation has also been applied to remote sensing [93] and medical data [94].”
- Continual Learning With Structured Inheritance for Semantic Segmentation in Aerial Imagery (2021, methodology): “Follow the prior works [24]–[26], [32], [33], [35], in order to avoid the loss of knowledge of the old tasks, we set the temperature τ of OPD loss to 1 and set the loss weight α in accordance with τ 2 .”
- Class-Incremental Learning for Semantic Segmentation Re-Using Neither Old Data Nor Old Labels (2020, methodology): “Previous approaches are either restricted to settings, where the additional classes have no overlap with the old ones [14], or rely on labels for both old and new classes [15].”
- ContCap: A scalable framework for continual image captioning (2019, background): “Rehearsal approaches [14], [15] demand data from old tasks, leading to privacy or limited storage budget issues, making them non-scalable if tasks arrive ceaselessly.”
- Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation (2021, background): “This setup, referred here as Continual Semantic Segmentation (CSS), has emerged very recently for specialized applications [11], [12], [13] before being proposed for general segmentation datasets [14], [15], [16].”
- ContCap: A comprehensive framework for continual image captioning (2019, background): “Rehearsal approaches [18][27][3] demand data from old tasks, leading to privacy or limited storage budget issues, making them non-scalable in the case of the number of classes is large.”
- CILEA-NET: Curriculum-Based Incremental Learning Framework for Remote Sensing Image Classification (2021, unknown): “In[26],theauthorsuseincrementallearningto performsemanticsegmentation;however,theoverallnumberof classesconsideredforthispurposeissmall.”
- CLRS: Continual Learning Benchmark for Remote Sensing Image Scene Classification (2020, methodology): “In the field of remote sensing, only Tasar et al. [17] applied the continual learning method to the semantic segmentation of remote sensing images.”
- Weakly-Supervised Continual Learning for Class-Incremental Segmentation (2022, methodology): “Following notation from [10], we refer through the rest of this paper to the previous network with the N − 1 class prediction as the memory network and to the new one as the updated network.”