NetTailor: Tuning the Architecture, Not Just the Weights

By Pedro Morgado, Nuno Vasconcelos. 2019

29 citations recorded by the source corpus.

Real-world applications of object recognition often require the solution of multiple tasks in a single platform. Under the standard paradigm of network fine-tuning, an entirely new CNN is learned per task, and the final network size is independent of task complexity. This is wasteful, since simple tasks require smaller networks than more complex tasks, and limits the number of tasks that can be solved simultaneously. To address these problems, we propose a transfer learning procedure, denoted NetTailor, in which layers of a pre-trained CNN are used as universal blocks that can be combined with small task-specific layers to generate new networks. Besides minimizing classification error, the new network is trained to mimic the internal activations of a strong unconstrained CNN, and minimize its complexity by the combination of 1) a soft-attention mechanism over blocks and 2) complexity regularization constraints. In this way, NetTailor can adapt the network architecture, not just its weights, to the target task. Experiments show that networks adapted to simple tasks, such as character or traffic sign recognition, become significantly smaller than those adapted to hard tasks, such as fine-grained recognition. More importantly, due to the modular nature of the procedure, this reduction in network complexity is achieved without compromise of either parameter sharing across tasks, or classification accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Class-Incremental Learning with Strong Pre-trained Models (2022, background): “The second line of work adopt dynamic models, where task-specific parameters are introduced to prevent interference among tasks, which can be achieved by growing the model [14,16,21,22,24,26,30,31,33], or introducing a gating function on the parameters [1, 18, 19].”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “This is often accomplished either by progressively growing the model each time a new task is added [ 37 , 68 , 82 ], or maintaining a common backbone model which is adapted to each task separately using small task-specific adaptation blocks [41, 51, 66].”
Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, background): “Existing continual learning works on tackling catastrophic forgetting can be broadly divided into three branches [10]: replay methods [1, 67], regularization-based methods [32, 47, 53, 85] and parameter isolation methods [55, 61, 70].”