Tree-CNN: A hierarchical Deep Convolutional Neural Network for incremental learning

By Deboleena Roy, Priyadarshini Panda, Kaushik Roy. 2018

242 citations recorded by the source corpus.

Over the past decade, Deep Convolutional Neural Networks (DCNNs) have shown remarkable performance in most computer vision tasks. These tasks traditionally use a fixed dataset, and the model, once trained, is deployed as is. Adding new information to such a model presents a challenge due to complex training issues, such as "catastrophic forgetting", and sensitivity to hyper-parameter tuning. However, in this modern world, data is constantly evolving, and our deep learning models are required to adapt to these changes. In this paper, we propose an adaptive hierarchical network structure composed of DCNNs that can grow and learn as new data becomes available. The network grows in a tree-like fashion to accommodate new classes of data, while preserving the ability to distinguish the previously trained classes. The network organizes the incrementally available data into feature-driven super-classes and improves upon existing hierarchical CNN models by adding the capability of self-growth. The proposed hierarchical model, when compared against fine-tuning a deep network, achieves significant reduction of training effort, while maintaining competitive accuracy on CIFAR-10 and CIFAR-100.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

Self-Supervised Training Enhances Online Continual Learning (2021, background): “There are three main approaches to mitigating catastrophic forgetting [53]: 1) increasing model capacity to include new representations that will not interfere with old knowledge [67, 60, 58, 3, 57, 54, 46, 45, 65], 2) regularizing continual parameter updates such that parameters do not deviate too ”
Knowledge Distillation for Incremental Learning in Semantic Segmentation (2019, methodology): “Focusing on deep neural networks, some methods [29, 30] exploit architectures which grow over time as a tree structure in a hierarchical manner as new classes are observed.”
Incremental Learning Techniques for Semantic Segmentation (2019, methodology): “Some methods [16, 27, 28, 36] exploit network architectures which grow during the training process.”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, background): “Recently, [46] propose an adaptive network that enables self-growth in a tree-like manner.”
Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing (2017, background): “[27] where the network grows in a tree-like manner to accommodate the new classes.”
Incremental Learning Using a Grow-and-Prune Paradigm With Efficient Neural Networks (2019, methodology): “also utilize such tree-structured networks to reduce the training cost by 20% [21].”
SPACE: Structured Compression and Sharing of Representational Space for Continual Learning (2020, background): “However, in the context of continual learning [4, 5, 6, 7], where humans can continuously adapt and learn new tasks throughout their lifetimes without forgetting the old ones, current deep learning methods struggle to perform well since the data distribution changes over the course of learning.”
Online Task-free Continual Learning with Dynamic Sparse Distributed Memory (2022, methodology): “Other methods of this kind dynamically add new layers to the network [51].”