Block Pruning For Faster Transformers

By François Lagunas, Ella Charlaix, Victor Sanh, Alexander M. Rush. 2021

267 citations recorded by the source corpus.

Pre-training has improved model accuracy for both classification and generation tasks at the cost of introducing much larger and slower models. Pruning methods have proven to be an effective way of reducing model size, whereas distillation methods are proven for speeding up inference. We introduce a block pruning approach targeting both small and fast models. Our approach extends structured methods by considering blocks of any size and integrates this structure into the movement pruning paradigm for fine-tuning. We find that this approach learns to prune out full components of the underlying model, such as attention heads. Experiments consider classification and generation tasks, yielding among other results a pruned model that is a 2.4x faster, 74% smaller BERT on SQuAD v1, with a 1% drop on F1, competitive both with distilled models in speed and pruned models in size.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Efficiency & Compression

Citations in the corpus

The Efficiency Misnomer (2021, background): “…Speed is one the most informative indicator for comparing the efﬁciency of different models (So et al., 2021; Dosovitskiy et al., 2020; Arnab et al., 2021; Tay et al., 2021b; Kim et al., 2021; He et al., 2021a; Narayanan et al., 2021; Lagunas et al., 2021; Liu et al., 2021; Tay et al., 2020c;b).”
Efficient Transformers: A Survey (2020, background): “[40] proposes a “block” pruning approach which can make a Transformer 2.”
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale (2022, unknown): “While they are many ways for model compression, such as quantization [38, 39, 40] and pruning [41, 42], our current efforts focus on layer reduction through knowledge distillation [43] (KD) – reducing both model size and model computation, and preserving MoE structure at student model.”
Learned Token Pruning for Transformers (2021, methodology): “Recent block pruning schemes chunk weight matrices into multiple blocks and prune them based on group Lasso optimization [26], adaptive regularization [53], and movement pruning [24].”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, methodology): “Subsequent methods [Li et al., 2016, Lin et al., 2017, Dong et al., 2017, Sanh et al., 2020, Lagunas et al., 2021, Zhu and Gupta, 2017] improve on the quality of the pruned models.”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, methodology): “…often overparameterized and can be pruned significantly with minimal impact on performance (Han et al., 2016; Wen et al., 2016; Liu et al., 2017; Luo et al., 2017; Cai et al., 2019; Deng et al., 2020; Hou et al., 2020; Wang et al., 2020; Lagunas et al., 2021; Xia et al., 2022; Kurtic et al., 2023).”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, background): “Lagunas et al. (2021) prunes transformers models with block sparsity pattern during fine-tuning, which leads to real hardware speed up while maintaining the accuracy.”
Intriguing Properties of Compression on Multilingual Models (2022, background): “…metrics in monolingual pre-trained LMs (Sanh et al., 2020; Goyal et al., 2020; Gordon et al., 2020; Budhraja et al., 2020; Sajjad et al., 2020; Lagunas et al., 2021; Xu et al., 2021; Du et al., 2021a; Ganesh et al., 2021), fewer works focus on multilingual settings (Mukherjee and Has-san…”
On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, methodology): “Speciﬁcally, DiﬀPruning (Mallya et al., 2018; Sanh et al., 2020; Guo et al., 2021; Lagunas et al., 2021) proposes to model the parameter selection mask as a Bernoulli random variable and optimize the variable with a reparametrization method.”
Structured Pruning Adapters (2022, methodology): “A group of fine-pruning methods [21, 23, 30, 40, 42, 45] have combined pruning with fine-tuning to reduce model size while learning parameters for the target task.”