Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing

By Syed Shakib Sarwar, Aayush Ankit, Kaushik Roy. 2017

118 citations recorded by the source corpus.

Deep convolutional neural network (DCNN) based supervised learning is a widely practiced approach for large-scale image classification. However, retraining these large networks to accommodate new, previously unseen data demands high computational time and energy requirements. Also, previously seen training samples may not be available at the time of retraining. We propose an efficient training methodology and incrementally growing DCNN to learn new tasks while sharing part of the base network. Our proposed methodology is inspired by transfer learning techniques, although it does not forget previously learned tasks. An updated network, for learning new set of classes, is formed using previously learned convolutional layers (shared from initial part of base network) with addition of few newly added convolutional kernels included in the later layers of the network. We employed a ‘clone-and-branch’ technique with calibration, which allows the network to learn new tasks (containing classes with similar features as old tasks) one after another without any performance loss in old tasks. We evaluated the proposed scheme on several recognition applications. The classification accuracy achieved by our approach is comparable to the regular incremental learning approach (where networks are updated with new training samples only, without any network sharing), while achieving energy efficiency, reduction in storage requirements, memory access and training time.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Gradient Projection Memory for Continual Learning (2021, background): “Dynamically Expandable Networks (DEN) (Yoon et al., 2018) either retrains or expands the network by splitting/duplicating important units on new tasks, whereas Sarwar et al. (2020) grow the network to learn new tasks while sharing part of the base network.”
Knowledge Distillation for Incremental Learning in Semantic Segmentation (2019, background): “In [19] the network incrementally grows over time while sharing portions of the base module.”
Incremental Learning Techniques for Semantic Segmentation (2019, methodology): “Some methods [16, 27, 28, 36] exploit network architectures which grow during the training process.”
Incremental Learning Through Deep Adaptation (2017, background): “[19] have proposed to train a network incre-”
Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data (2018, methodology): “In [16], the network is trained incrementally by sharing early layers and splitting later ones by adding new convolutional kernels.”
Tree-CNN: A hierarchical Deep Convolutional Neural Network for incremental learning (2018, background): “It has been observed that initial layers of a CNN learn very generic features [17] that has been exploited for transfer learning [18]„ [19].”
Continual Learning Using Data Regeneration for Remote Sensing Scene Classification (2022, background): “Sarwar et al. [9] proposed to expand a trained network with additional convolution layers related to the new tasks.”
Seer: Leveraging Big Data to Navigate the Complexity of Performance Debugging in Cloud Microservices (2019, methodology): “To adjust to changes in the execution environment, Seer retrains incrementally in the background, using the transfer learning -based approach in [80].”
Continual Learning with Filter Atom Swapping (2022, background): “On the other hand, storing part of parameters, or small subsets of data for parameter ﬂashback cannot avoid forgetting completely (Sarwar et al., 2019; Yoon et al., 2018).”
Continual Learning for Dense Labeling of Satellite Images (2019, methodology): “An example methodology that is in this category is described in [3], where the network is expanded as new classes are added.”
Controllable Semantic Parsing via Retrieval Augmentation (2021, background): “Existing work commonly tackles catastrophic forgetting via incremental training, such as imposing constraints on the distance between new and old models (Sarwar et al., 2019; Rosenfeld and Tsotsos, 2018) or jointly learn a generator to reply past examples for training (Hu et al.”
SPACE: Structured Compression and Sharing of Representational Space for Continual Learning (2020, background): “pruning (PackNet) [21], growing [24], [31]–[35], activation masking (HAT) [26] or introducing parallel paths [22], [36] for new tasks.”