RILOD: near real-time incremental learning for object detection at the edge

By Dawei Li, Serafettin Tasci, Jingwen Zhu, Junting Zhang, Larry Heck. 2019

114 citations recorded by the source corpus.

Object detection models shipped with camera-equipped edge devices cannot cover the objects of interest for every user. Therefore, the incremental learning capability is a critical feature for a robust and personalized object detection system that many applications would rely on. In this paper, we present an efficient yet practical system, RILOD, to incrementally train an existing object detection model such that it can detect new object classes without losing its capability to detect old classes. The key component of RILOD is a novel incremental learning algorithm that trains end-to-end for one-stage deep object detection models only using training data of new object classes. Specifically to avoid catastrophic forgetting, the algorithm distills three types of knowledge from the old model to mimic the old model's behavior on object classification, bounding box regression and feature extraction. In addition, since the training data for the new classes may not be available, a real-time dataset construction pipeline is designed to collect training images on-the-fly and automatically label the images with both category and bounding box annotations. We have implemented RILOD under both edge-cloud and edge-only setups. Experiment results show that the proposed system can learn to detect a new object class in just a few minutes, including both dataset construction and model training. In comparison, traditional fine-tuning based method may take a few hours for training, and in most cases would also need a tedious and costly manual dataset labeling step.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “The idea of knowledge distillation is then introduced to other detection frameworks, such as CenterNet [508] (SID [328]), RetinaNet [259] (RILOD [247]), GFLV1 [250] (ERD [114]) and Faster R-CNN [358] (CIFRCN [152], Faster ILOD [327], DMC [493], BNC [96], IOD-ML [222]).”
An EM Framework for Online Incremental Learning of Semantic Segmentation (2021, neutral): “This assumption, while simplifying the task formulation, can be restrictive in real-world applications, such as medical image analysis, robot sensing, and personalized mobile Apps [18], where novel visual concepts need to be incorporated later on or continuously in deployment.”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, background): “Continual learning has been widely studied in image classification [32, 36] and object detection [56, 34], while has been tackled only recently in the semantic segmentation field [42, 58, 4, 33].”
Towards Open World Object Detection (2021, background): “The prevalent approaches [61, 28, 18, 7] use knowledge distillation [21]”
Online Continual Learning in Image Classification: An Empirical Survey (2021, methodology): “Most methods leverage KD [39] to alleviate CF, and the main differences between the methods are the base object detector and distillation parts in the network [125, 126, 127, 128].”
SID: Incremental Learning for Anchor-Free Object Detection via Selective and Inter-Related Distillation (2020, methodology): “Li et al. (2019) and Zhang et al. (2020) designed incremen-2 tal detectors based on RetinaNet (Lin et al., 2017).”
Incremental Object Detection via Meta-Learning (2020, positive): “To this end, knowledge distillation [5] has been utilized as a de facto solution [6], [7], [8], [9].”
Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN (2020, methodology): “(Li et al., 2019) proposed an one-stage incremental object detector called RILOD based on RetinaNet (Lin et al., 2017).”
Latent Replay for Real-Time Continual Learning (2019, methodology): “Finally, training on the edge was recently addressed in [1], where an object detection model was incrementally trained based on LWF and pattern replay.”
RODEO: Replay for Online Object Detection (2020, neutral): “Similarly, [29] uses distillation on the classification predictions, bounding box coordinates, and network features to train an end-to-end incremental network.”
Incremental Detection of Remote Sensing Objects With Feature Pyramid and Knowledge Distillation (2020, methodology): “Based on one-stage methods, [22] distills three types of knowledge from the frozen model to mimic its behavior on classiﬁcation, regression, and feature extraction.”
Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection (2023, methodology): “Relatively few works [32, 49, 53] used one-stage architectures for incremental learning.”
Online Continual Learning for Embedded Devices (2022, background): “Most focus on supervised classification using small batches on the CORe50 dataset (Lomonaco & Maltoni, 2017) with a MobileNet-v1 backbone (Pellegrini et al., 2020; 2021; Demosthenous & Vassiliades, 2021), while one focuses on continual object detection (Li et al., 2019).”
Class-Incremental Learning: A Survey (2023, background): “There are several ways to expand representation ability, and we divide them into three sub-groups, i.e. , neuron expansion , backbone expansion , and prompt expansion .”
Continual Detection Transformer for Incremental Object Detection (2023, methodology): “Table S1 shows that also with the traditional IOD protocol our CL-DETR consistently performs better than the state-of-the-art [13] and other IOD methods [25, 27, 40].”
IncDet: In Defense of Elastic Weight Consolidation for Incremental Object Detection (2020, methodology): “RILOD [22] introduces an extra feature distillation loss besides the classification and regression distillation loss in ILOD [8], and it is implemented under both edge-cloud and edge-only setups.”
SDDGR: Stable Diffusion-Based Deep Generative Replay for Class Incremental Object Detection (2024, methodology): “CIOD has progressed from primarily employing CNN-based methods [1, 11, 28, 36, 43, 60] to also incorporating Transformer-based approaches [10, 15, 24, 25, 37].”