Growing a Brain: Fine-Tuning by Increasing Model Capacity

By Yu-Xiong Wang, Deva Ramanan, Martial Hebert. 2017

164 citations recorded by the source corpus.

CNNs have made an undeniable impact on computer vision through the ability to learn high-capacity models with large annotated training sets. One of their remarkable properties is the ability to transfer knowledge from a large source dataset to a (typically smaller) target dataset. This is usually accomplished through fine-tuning a fixed-size network on new target data. Indeed, virtually every contemporary visual recognition system makes use of fine-tuning to transfer knowledge from ImageNet. In this work, we analyze what components and parameters change during fine-tuning, and discover that increasing model capacity allows for more natural model adaptation through fine-tuning. By making an analogy to developmental learning, we demonstrate that growing a CNN with additional units, either by widening existing layers or deepening the overall network, significantly outperforms classic fine-tuning approaches. But in order to properly grow a network, we show that newly-added units must be appropriately normalized to allow for a pace of learning that is consistent with existing units. We empirically validate our approach on several benchmark datasets, producing state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Scaling Laws for Neural Language Models (2020, methodology): “And using methods like [WRH17, WYL19], which grow networks as they train, it might be possible to remain on the compute-efﬁcient frontier for an entire training run.”
Learning without Forgetting (2016, background): “We note that Growing a Brain [29] offers a more thorough experiment of network expansion (w/o LwF).”
Knowledge Flow: Improve Upon Your Teachers (2019, methodology): “…transfer have been considered using a variety of techniques, for instance, ﬁne-tuning, progressive neural nets (Rusu et al., 2016b), PathNet (Fernando et al., 2017), ‘Growing a Brain’ (Wang et al., 2017), actor-mimic (Parisotto et al., 2016), learning without forgetting (Li & Hoiem, 2016).”
Dota 2 with Large Scale Deep Reinforcement Learning (2019, methodology): “Past methods have grown neural networks by incrementally training and freezing parts of the network [49], [50], [51].”
Continuous Learning in Single-Incremental-Task Scenarios (2018, background): “rate and the representation cannot be further adapted to new data. To deal with this problem we believe that the incremental lateral expansion of some intermediate layers, similar to that proposed in [32] can be a valid approach. Class-incremental learning (NC update content type) is only one of the”
Class-Incremental Learning with Strong Pre-trained Models (2022, background): “We differ from prior network-growing approaches [24, 26] in two ways.”
RECALL: Replay-based Continual Learning in Semantic Segmentation (2021, background): “Dynamic architectures can be explicit [42, 26], if new network branches are grown, or implicit [13, 37], if some network weights are available for certain tasks only.”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “There are three main approaches to mitigating catastrophic forgetting [53]: 1) increasing model capacity to include new representations that will not interfere with old knowledge [67, 60, 58, 3, 57, 54, 46, 45, 65], 2) regularizing continual parameter updates such that parameters do not deviate too ”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, background): “Dynamic architectures [64, 35] grow new branches for new tasks.”
Memory-Efficient Incremental Learning Through Feature Adaptation (2020, background): “pdating the model with new classes [33]. Non-rehearsal methods do not preserve any data from previously seen classes. Common approaches include increasing the network capacity for new sets of classes [35,38], or weight consolidation, which identies the important weights for previous sets of classes ”
ScaIL: Classifier Weights Scaling for Class Incremental Learning (2020, background): “In a first scenario [2, 18, 28, 30], the number of deep model parameters is allowed to grow and no memory is used.”
IL2M: Class Incremental Learning With Dual Memory (2019, background): “A stream of research [1, 17, 27, 29] assumes that deep architectures can grow to some extent so as to integrate new data.”
Continual Learning for Robotics (2019, background): “Instead, layers may be dynamically expanded in a single network without the need of re-training or freezing previously learned parameters, improving model capacity over time (Wang et al., 2017).”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, background): “While some continual learning works consider the problem of forward transfer, e.g. Rusu et al. (2016); Aljundi et al. (2017); Wang et al. (2017), these works and others in continual learning generally focus on small sets of tasks where fast, online learning is not realistically possible, since…”
Online Meta-Learning (2019, neutral): “Furthermore, unlike prior works (Ruvolo & Eaton, 2013; Rusu et al., 2016; Aljundi et al., 2017; Wang et al., 2017), we also focus on the setting where there are several tens or hundreds of tasks.”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, methodology): “C om pl ex ity The model evolves by adding parameters and weights to interconnect them [42, 48, 56] or small networks [2] to include new knowledge.”
DeeSIL: Deep-Shallow Incremental Learning (2018, background): “ood data representation when augmenting the number of tasks. However, it violates the third property of IL algorithms since its number of parameters increases with the number of tasks. The authors of [5] and [6] improve the plasticity of deep architectures by widening existing layers and/or deepenin”
Exploring the Challenges towards Lifelong Fact Learning (2018, background): “ ImageNet. This assumption also applies to systems that learn concepts by reading the web (e.g., [20,5,6]) or that augment CNNs with additional units to better transfer knowledge to new tasks such as [29]. To get closer to human visual learning and to practical application scenarios, where data ofte”
Learning to Model the Tail (2017, methodology): “We first focus on fine-tuning the classifier module while freezing the representation module of a pre-trained ResNet152 CNN model [4, 63] for its state-of-the-art performance.”
Continual Backprop: Stochastic Gradient Descent with Persistent Randomness (2021, background): “Multiple recent works have focused on adapting to the changes in the data stream by ﬁnding new useful information (Rusu et al. 2016, Finn et al. 2017, Wang et al. 2017; Nagabandi et al. 2019, Finn et al. 2019).”