Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture

By Christopher Brix, Parnia Bahar, Hermann Ney. 2020

43 citations recorded by the source corpus.

Sparse models require less memory for storage and enable a faster inference by reducing the necessary number of FLOPs. This is relevant both for time-critical and on-device computations using neural networks. The stabilized lottery ticket hypothesis states that networks can be pruned after none or few training iterations, using a mask computed based on the unpruned converged model. On the transformer architecture and the WMT 2014 English-to-German and English-to-French tasks, we show that stabilized lottery ticket pruning performs similar to magnitude pruning for sparsity levels of up to 85%, and propose a new combination of pruning techniques that outperforms all other techniques for even higher levels of sparsity. Furthermore, we confirm that the parameter’s initial sign and not its specific value is the primary factor for successful training, and show that magnitude pruning cannot be used to find winning lottery tickets.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Efficiently Scaling Transformer Inference (2022, neutral): “…2019; Child et al., 2019), distillation (Sanh et al., 2019; Sun et al., 2020), and model compression techniques, such as pruning (Li et al., 2020b; Brix et al., 2020; Zhou et al., 2021; Li et al., 2020a; Wang et al., 2020), or quantization (Dettmers et al., 2022; Abdolrashidi et al., 2021;…”
Sparse is Enough in Scaling Transformers (2021, critical): “Model pruning [24, 2] makes matrices smaller by removing unneeded weights after or during training, however, the gains in computational complexity on sparse matrices often do not result in inference speedups on actual hardware [9].”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, background): “While it is exciting to see widespread applicability of LTs in different domains (Brix et al., 2020; Li et al., 2020; Venkatesh et al., 2020), the results presented in this paper suggest this success may be due to the underlying pruning algorithm (and transfer learning) rather than LT initialization”
Continual Pre-training of Language Models (2023, methodology): “Existing methods include discarding parameters with small absolute values (Han et al., 2015; Guo et al., 2016), accumulated gradient (Michel et al., 2019), and lottery ticket hypothesis (Brix et al., 2020).”
High Performance Natural Language Processing (2020, background): “(2019); Frankle and Carbin (2018); Brix et al. (2020); Sanh et al. (2020).”