TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning

By Han Cai, Chuang Gan, Ligeng Zhu, Song Han. 2020

359 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “On-Device Learning On-device AI is becoming more ubiquitously adopted to enable model personalization [84], [85], [86] while improving data privacy [87], [88], [89], [90], yet its impact in terms of carbon emission is often overlooked.”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “[33] propose to freeze the weights and only train the biases.”
Visual Prompt Tuning (2022, background): “– Bias [6,79]: fine-tune only the bias terms of a pre-trained backbone.”
A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays (2021, methodology): “TinyTL [22] demonstrated on a high-end CPU that the transfer learning task results more effective (+32% on the target Image Classification task) by retraining the bias terms and adding lite residual modules.”
Improving Visual Prompt Tuning for Self-supervised Vision Transformers (2023, background): “Cai et al. (2020) proposed to freeze the weights and only update the bias of the pre-trained models.”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, neutral): “Cai et al. [111] proposed another way to reduce the memory required for on-device training by introducing a lite residual module that can be adapted to new data.”
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, methodology): “Existing PETL methods, such as Adapters [23], LoRA [24], Prompt-tuning [31], and BitFit [58, 3]”
6G Internet of Things: A Comprehensive Survey (2021, methodology): “To bridge this gap, we have ﬁrst introduced the recent TABLE VI: Summary of key research challenges and possible directions for 6G-IoT.”
Federated Learning for Internet of Things: A Comprehensive Survey (2021, methodology): “Additionally, a scheme called Tiny-TransferLearning (TinyTL) is proposed in [206] for memory-efficient on-device sensor learning.”
Prompt-free and Efficient Few-shot Learning with Language Models (2022, methodology): “Following Pfeiffer et al. (2021), we experimented with keeping one of the adapters in each layer for better training efficiency and found keeping the adapter after the feed-forward module in each layer to perform the best.”
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “Compared to STN, AdaIN, FiLM and so on, our method is input-independent and these scale and shift parameters model the distribution of the whole dataset so that they can be absorbed into the original pre-trained model weights in the inference phase.”
Differentially Private Fine-tuning of Language Models (2021, background): “…dimensionality (Li et al., 2018; Aghajanyan et al., 2020)), preﬁx and prompt tuning (Li & Liang, 2021; Lester et al., 2021), tuning only biases (Cai et al., 2020; Ben Zaken et al., 2021), and other architecture variants including Adapters (Pfeiffer et al., 2021; R ¨ uckl ´ e et al., 2020).”
Large Language Models Can Be Strong Differentially Private Learners (2021, methodology): “In addition, there are adaptation methods that only optimize biases for vision (Cai et al., 2020) and language tasks (Ben Zaken et al.”