K For The Price Of 1: Parameter Efficient Multi-task And Transfer Learning

By Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, Andrew G. Howard. 2018

76 citations recorded by the source corpus.

We introduce a novel method that enables parameter-efficient transfer and multi-task learning with deep neural networks. The basic approach is to learn a model patch - a small set of parameters - that will specialize to each task, instead of fine-tuning the last layer or the entire network. For instance, we show that learning a set of scales and biases is sufficient to convert a pretrained network to perform well on qualitatively different problems (e.g. converting a Single Shot MultiBox Detection (SSD) model into a 1000-class image classification model while reusing 98% of parameters of the SSD feature extractor). Similarly, we show that re-learning existing low-parameter layers (such as depth-wise convolutions) while keeping the rest of the network frozen also improves transfer-learning accuracy significantly. Our approach allows both simultaneous (multi-task) as well as sequential transfer learning. In several multi-task learning problems, despite using much fewer parameters than traditional logits-only fine-tuning, we match single-task performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, background): “The ﬁrst baseline is “BN-Tuned”, which ﬁne-tunes batch normalization parameters per task and which has previously shown strong performance for multi-task learning (Mudrakarta et al., 2018).”
Neural networks with late-phase weights (2020, background): “…additional parameters have large expressive power: it has been shown that learning only γ and β keeping the remaining weights frozen can lead to significantly lower loss than when learning random subsets of other weights of matching dimensionality (Frankle et al., 2020; Mudrakarta et al., 2019).”
Fine-tuning Image Transformers using Learnable Memory (2022, background): “During fine-tuning, only those layers and/or normalization parameters [30, 44] are updated.”
Attentive Single-Tasking of Multiple Tasks (2019, background): “Several works study modulation for image retrieval [69] or classiﬁ-cation tasks [44, 40], and embeddings for different artistic styles [11].”
On-Device Training Under 256KB Memory (2022, background): “Later studies investigate other tuning methods including updating biases [13, 71], updating normalization layer parameters [54, 26], updating small parallel branches [13, 33], etc.”
TranSlider: Transfer Ensemble Learning from Exploitation to Exploration (2020, background): “Yet more recent studies [11, 16, 26, 27, 33, 41] attempt to simultaneously learn a transfer strategy and the parameters for the target model.”
FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification (2022, methodology): “…parameter efﬁcient adapters have been proposed including FiLM, Adapter (Houlsby et al., 2019), LoRA (Hu et al., 2021), VPT (Jia et al., 2022), AdaptFormer (Chen et al., 2022), NOAH (Zhang et al., 2022), Convpass (Jie & Deng, 2022), (Mudrakarta et al., 2019), and CaSE (Patacchiola et al., 2022).”
TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning (2020, methodology): “Recently, [45,46] propose to only update parameters of the batch normalization (BN) [47] layers, which greatly reduces the number of trainable parameters.”
Tiny Transfer Learning: Towards Memory-Efficient On-Device Learning (2020, background): “However, the capacity of this approach is limited, resulting in poor accuracy, especially on datasets [30,35] whose distribution is far from ImageNet (e.g., only 45.9% Aircraft top1 accuracy achieved by Inception-V3 [36]).”