Side-Tuning: Network Adaptation via Additive Side Networks

By Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, Jitendra Malik. 2019

29 citations recorded by the source corpus.

When training a neural network for a desired task, one may prefer to adapt a pre-trained network rather than start with a randomly initialized one -- due to lacking enough training data, performing lifelong learning where the system has to learn a new task while being previously trained for other tasks, or wishing to encode priors in the network via preset weights. The most commonly employed approaches for network adaptation are fine-tuning and using the pre-trained network as a fixed feature extractor, among others. In this paper, we propose a straightforward alternative: Side-Tuning. Side-tuning adapts a pre-trained network by training a lightweight "side" network that is fused with the (unchanged) pre-trained network using summation. This simple method works as well as or better than existing solutions while it resolves some of the basic issues with fine-tuning, fixed features, and several other common baselines. In particular, side-tuning is less prone to overfitting when little training data is available, yields better results than using a fixed feature extractor, and does not suffer from catastrophic forgetting in lifelong learning. We demonstrate the performance of side-tuning under a diverse set of scenarios, including lifelong learning (iCIFAR, Taskonomy), reinforcement learning, imitation learning (visual navigation in Habitat), NLP question-answering (SQuAD v2), and single-task transfer learning (Taskonomy), with consistently promising results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Meta-Consolidation for Continual Learning (2020, background): “Note that ‘task’ here can refer to a set of new classes, new domains (e.g. thermal, RGB) or even new tasks in general (e.g. colorization, segmentation) [62, 67, 88].”
The Effectiveness of Memory Replay in Large Scale Continual Learning (2020, methodology): “Our proposed approach differs from Side-Tuning in that we allow the representation network to be shared and evolved over tasks with stable model capacity and inference time.”
Efficient Feature Transformations for Discriminative and Generative Continual Learning (2021, methodology): “A number of previous works [66, 59, 43, 57, 8, 45, 29, 30, 51, 58] have proposed methods for expanding a neural network to learn a sequence of tasks.”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, methodology): “Side-Tuning [91] uses a side branch model to learn extra functionality by linearly blending the outputs of a frozen model and an added network, with a predefined blending weight schedule.”
Bayesian Nonparametric Weight Factorization for Continual Learning (2020, methodology): “Side-tuning [47] learns a separate “side” network for each task, adding the output to a shared base model; while this experiences linear growth T of the model size, it reduces the cost by keeping each side network small.”
Residual Tuning: Toward Novel Category Discovery Without Labels (2022, unknown): “Two recent works similar to ours are ResCL [61] and Sidetuning [22], both of which were focused on continual learning.”
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, methodology): “[53] found that late fusion to combine the representations of the backbone and the side network works well with convolutional networks for CV tasks, in our experiments, we find that late fusion hurts the performance of the transformer architecture in NLP tasks (see Figure 8 in Section 5 for details)”