Incremental Learning of Object Detectors without Catastrophic Forgetting

By Konstantin Shmelkov, Cordelia Schmid, Alahari Karteek. 2017

607 citations recorded by the source corpus.

Despite their success for object detection, convolutional neural networks are ill-equipped for incremental learning, i.e., adapting the original model trained on a set of classes to additionally detect objects of new classes, in the absence of the initial training data. They suffer from “catastrophic forgetting”–an abrupt degradation of performance on the original set of classes, when the training objective is adapted to the new classes. We present a method to address this issue, and learn object detectors incrementally, when neither the original training data nor annotations for the original classes in the new training set are available. The core of our proposed solution is a loss function to balance the interplay between predictions on the new classes and a new distillation loss which minimizes the discrepancy between responses for old classes from the original and the updated networks. This incremental learning can be performed multiple times, for a new set of classes in each step, with a moderate drop in performance compared to the baseline network trained on the ensemble of data. We present object detection results on the PASCAL VOC 2007 and COCO datasets, along with a detailed empirical analysis of the approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Note that, ak,j can be adapted to other forms depending on the task type, such as average precision (AP) for object detection [384], Intersection-over-Union (IoU) for semantic segmentation [56], Fréchet Inception Distance (FID) for image generation [452], normalized reward for reinforcement learning”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, methodology): “…our results are based on a pre-trained ViT backbone for a fair comparison with previous works [45, 44], without considering other representative architectures such as ResNet and other CL applications on downstream computer vision tasks such as object detection [34], image segmentation [53, 54, 51].”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, background): “In this regard, the mitigation of catastrophic forgetting [52] has driven the recent research towards novel incremental methods [42, 68], often framed under the field of Continual Learning (CL).”
Learning with Recoverable Forgetting (2022, neutral): “Apart from those works that still need part of the old data, many researchers are devoted to developing the methods without storing the old data by synthesizing old data [9,41,46] or even without referring to any old data [44,42,32].”
EPicker is an exemplar-based continual learning approach for knowledge accumulation in cryoEM particle picking (2022, methodology): “Knowledge distillation 25 is a widely used technique for incremental object detection problems in natural images 26 – 28 , which transfers knowledge between neural networks using loss functions that minimize the difference between features extracted from the new and old models for old datasets.”
Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation (2022, methodology): “[31] proposed to apply LwF to Fast RCNN detector [10], which is the first work on incremental object detection.”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “Following previous works [17], [19], [20], [23], we propose two different experimental settings: disjoint and overlapped.”
Federated Class-Incremental Learning (2022, methodology): “Without access to the data of old classes, [20] designs new regulators for balancing the biased model optimization caused by new classes, and [25,41] use the knowledge distillation to surmount catastrophic forgetting.”
Memory Efficient Continual Learning with Transformers (2022, methodology): “[52] propose an end-to-end learning framework where the representation and the classiﬁer are learned jointly without storing any of the original training samples.”
Static-Dynamic Co-Teaching for Class-Incremental 3D Object Detection (2021, methodology): “Although class-incremental learning has been studied in several computer vision tasks (Li and Hoiem 2017; Shmelkov, Schmid, and Alahari 2017; Michieli and Zanuttigh 2019; Dong et al. 2021), especially image classification, classincremental learning of 3D object detection remains unexplored.”
Bridging Non Co-occurrence with Unlabeled In-the-wild Data for Incremental Object Detection (2021, neutral): “Furthermore, we achieve higher performances compared to [27] (Ours: 70.”
Class-Incremental Learning for Action Recognition in Videos (2021, neutral): “Although class-incremental learning has been studied for image classiﬁcation, the research is also active for other applications, including person re-identiﬁcation [42], 3D object classiﬁcation [5], object detection [33], and semantic segmentation [24].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “To solve this problem, they proposed to separate parameters into global and task-specific parameters, whereby each client can obtain a weighted combination of the task-specific parameters of the other clients for knowledge transfer.”
Wanderlust: Online Continual Object Detection in the Real World (2021, methodology): “In the object detection domain, several incremental object detection algorithms [23, 27, 32, 41] adopt existing object detection datasets such as PASCAL VOC [11] and MS COCO [26] for evaluation.”
Continual semi-supervised learning through contrastive interpolation consistency (2021, background): “However, when the labeling stage involves human intervention (as holds in a number of computer vision tasks such as classification, object detection [5, 6], etc.”
RECALL: Replay-based Continual Learning in Semantic Segmentation (2021, methodology): “Following previous works on this topic [40, 29, 31, 5], we start by analyzing the performance on three widely used incremental scenarios: i.”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, background): “Despite the progress on image classiﬁcation, recent works further investigate continual learning on object detection [37] and semantic segmentation [2] with knowledge distillation.”
Optimizing Reusable Knowledge for Continual Learning via Metalearning (2021, background): “In the case of artificial neural networks (ANNs), task interference is behind the problem known as Catastrophic Forgetting (CF) [3, 4, 5, 6, 7].”
Continual Learning for Named Entity Recognition (2021, methodology): “Previous work mostly focused on Computer Vision (Shmelkov, Schmid, and Alahari 2017; Li and Hoiem 2018; Rannen et al. 2017) by using Knowledge Distillation (Hin-ton, Vinyals, and Dean 2015) as the base framework.”