Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks

By Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, Jitendra Malik. 2019

298 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

Visual Prompt Tuning (2022, background): “(c) Methods that update a subset backbone parameters or add new trainable parameters to backbone during fine-tuning: – Sidetune [87]: train a “side” network and linear interpolate between pre-trained features and side-tuned features before being fed into the head.”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “As for adapting a ﬁxed backbone, Side-Tuning [198] and DLCFT [199] train a lightweight network in parallel with the backbone and fuse their outputs linearly.”
Class-Incremental Learning with Strong Pre-trained Models (2022, background): “The second line of work adopt dynamic models, where task-specific parameters are introduced to prevent interference among tasks, which can be achieved by growing the model [14,16,21,22,24,26,30,31,33], or introducing a gating function on the parameters [1, 18, 19].”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “Early works for efficient transfer learning in vision focus on parameter sharing in the context of multitask learning [83, 57, 56].”
Computationally Budgeted Continual Learning: What Does Matter? (2023, methodology): “This is done by only training part of the model [1, 4, 39, 40, 44] or by directly expanding the model when data is presented [46, 49, 54, 57, 59, 62].”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, background): “Recent work has demonstrated that MTL methods developed on existing datasets seem to specialize to their respective developement set and do not perform well on large, realistic datasets, or on other tasks [62, 63, 73].”
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (2022, unknown): “We also compare with another method, (3) side-tuning (Zhang et al., 2020).”
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, methodology): “Parameter-efﬁcient transfer learning (PETL) [31, 23, 39, 40, 34, 51, 58, 17, 24, 18, 42, 60, 14, 52, 59] is a recent research direction for online or multi-task learning.”
A Survey on Negative Transfer (2020, background): “[125] Co-Tuing Deep TL/Model adaptation D, T, A Model diversity”
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “Here, we ﬁnetune a 220M T5-Base model on top of the outputs generated from the PaLM model, which can be viewed as a ﬁnetuning method for LLMs with additional parameters (Zhang et al., 2020; Lester et al., 2021).”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “1 [12], QDyLoRA [150], BitDelta [118] Memory-efficientPEFT Side-Tuning [213], LST [167], Res-Tuning [85], MEFT [112], LoRA-FA [215], HyperTuning [146], PEFT Plug-in [88], MeZO [131], GaLore [228] Fig.”
Side Adapter Network for Open-Vocabulary Semantic Segmentation (2023, background): “The most related works to us are Side-Tuning [38] and its variants [34], a side network is attached to the pre-training model and the final representation is a combination of the side network and the pre-training model.”
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2021, methodology): “There are various ways to parameterize the task-specific differences δtask, leading to different additive methods such as LoRA (Hu et al., 2021), diff pruning (Guo et al., 2021), and side-tuning (Zhang et al., 2020).”
Factorizing Knowledge in Neural Networks (2022, background): “Some recent solutions explore on the decomposition between shared and task-speciﬁc processing [40,25,69].”