Training Neural Networks with Fixed Sparse Masks

By Yi-Lin Sung, Varun Nair, Colin Raffel. 2021

282 citations recorded by the source corpus.

During typical gradient-based training of deep neural networks, all of the model's parameters are updated at each iteration. Recent work has shown that it is possible to update only a small subset of the model's parameters during training, which can alleviate storage and communication requirements. In this paper, we show that it is possible to induce a fixed sparse mask on the model's parameters that selects a subset to update over many iterations. Our method constructs the mask out of the $k$ parameters with the largest Fisher information as a simple approximation as to which parameters are most important for the task at hand. In experiments on parameter-efficient transfer learning and distributed training, we show that our approach matches or exceeds the performance of other methods for training with sparse updates while being more efficient in terms of memory usage and communication costs. We release our code publicly to promote further applications of our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “…(PEFT) methods have been proposed such as prompt tuning [49, 33, 34], tuning the embedding layer inputs [1], tuning hidden states (IA 3 ) [37], adding full layers [27], tuning biases [71], learning a mask over weights based on Fisher information [55], and a combination of approaches [23].”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Specifically, we compare with 9 strong PEFT methods: BitFit [47] which updates only the bias parameters; Adapters [23] which introduce task-specific layers after the self-attention and position-wise feed-forward networks; Compacter and Compacter++ [28] which improve upon adapters by using low-rank m”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “2 To address this problem, recently, several parameter-efficient training methods have been proposed [12, 16, 17, 20, 28, 33, 47, 52].”
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022, methodology): “Note that the importance matrix I used in this work are either weight magnitude [33] ( I = W ) or empirical Fisher Information [51] ( I = F W = 1 N =1 ( ∇ W log p ( y i | x i )) 2 ; ( x i , y i ) , ..., ( x N , y N ) are samples from data).”
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, unknown): “We experiment with (IA) 3 since it outperforms bottleneck adapters, which are used in MAD-X, and other parameter-efficient finetuning methods such as BitFit (Ben Zaken et al., 2022), LoRA (Hu et al., 2022), and FishMask (Sung et al., 2021) on English NLU tasks (Liu et al., 2022 than these methods (s”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, unknown): “PEFT encompasses strategies like partial fine-tuning [15, 16, 17, 18, 19, 20, 21, 22], soft prompt fine-tuning [23, 24, 25, 26, 27, 28, 29], non-linear adapter fine-tuning [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], and low rank adapter based fine-tuning [40, 41, 11, 42].”
Editing Models with Task Arithmetic (2022, unknown): “Considering that re-training models is prohibitively expensive in most circumstances, several authors have studied more efficient methods for modifying a model’s behavior with interventions after pre-training, referring to this process by different names, such as patching [33; 89; 39; 71], editing [”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “…the bias term in the attention layer, feed-forward layer and layer normalization of the transformer is updated, Pretrained Weight Masking [36], [37], where the pretrained weights are masked using various pruning criterion, and Delta Weight Masking [38], [39], [40], [41], in which delta weights…”
Git Re-Basin: Merging Models modulo Permutation Symmetries (2022, background): “Ul-timately, we anticipate that a lucid understanding of loss landscape geometry will not only advance the theory of deep learning but will also promote the development of better optimization, federated learning, and ensembling techniques.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “U-Diff pruning [59], U-BitFit [99], PaFi [111], FishMask [169], Fish-Dip [34], LT-SFT [3], SAM [48], Child-tuning [194] Structural Masking S-Diff pruning [59], S-BitFit [99], FAR [174], Bitfit [207], Xattn Tuning [54], SPT [66] ReparameterizedFine-tuning”
On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, methodology): “These methods are found to be more stable than fine-tuning the full model (He et al., 2021b; Lee et al., 2019; Houlsby et al., 2019; Zaken et al., 2021; Sung et al., 2021; Liu et al., 2021).”