Incremental Training of Deep Convolutional Neural Networks
By Roxana Istrate, A. Malossi, C. Bekas, Dimitrios S. Nikolopoulos. 2018
21 citations recorded by the source corpus.
We propose an incremental training method that partitions the original network into sub-networks, which are then gradually incorporated in the running network during the training process. To allow for a smooth dynamic growth of the network, we introduce a look-ahead initialization that outperforms the random initialization. We demonstrate that our incremental approach reaches the reference network baseline accuracy. Additionally, it allows to identify smaller partitions of the original state-of-the-art network, that deliver the same final accuracy, by using only a fraction of the global number of parameters. This allows for a potential speedup of the training time of several factors. We report training results on CIFAR-10 for ResNet and VGGNet.
Subfields: Artificial Intelligence, Control and Systems Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Pietro Zanuttigh
- Umberto Michieli
- Hongxu Yin
- Niraj Kumar Jha
- Xiaoliang Dai
- C. Bekas
- Giovanni Mariani
- Dimitrios S. Nikolopoulos
In Control and Systems Engineering
Citations in the corpus
- TAPAS: Train-less Accuracy Predictor for Architecture Search (2018, methodology): “To speed up the process, we train the networks one layer at a time using the incremental method described in (Istrate et al. 2017).”
- Knowledge Distillation for Incremental Learning in Semantic Segmentation (2019, methodology): “[18] proposed a method that partitions the original network into sub-networks which are then gradually incorporated in the main one during training.”
- Incremental Learning Techniques for Semantic Segmentation (2019, methodology): “Some methods [16, 27, 28, 36] exploit network architectures which grow during the training process.”
- Incremental Learning Using a Grow-and-Prune Paradigm With Efficient Neural Networks (2019, methodology): “%) training cost reduction compared to the conventional TFS (NFT) approach on LeNet-300-100.”