When BERT Plays the Lottery, All Tickets Are Winning

By Sai Prasanna, Anna Rogers, Anna Rumshisky. 2020

206 citations recorded by the source corpus.

Much of the recent success in NLP is due to the large Transformer-based models such as BERT (Devlin et al, 2019). However, these models have been shown to be reducible to a smaller number of self-attention heads and layers. We consider this phenomenon from the perspective of the lottery ticket hypothesis. For fine-tuned BERT, we show that (a) it is possible to find a subnetwork of elements that achieves performance comparable with that of the full model, and (b) similarly-sized subnetworks sampled from the rest of the model perform worse. However, the "bad" subnetworks can be fine-tuned separately to achieve only slightly worse performance than the "good" ones, indicating that most weights in the pre-trained BERT are potentially useful. We also show that the "good" subnetworks vary considerably across GLUE tasks, opening up the possibilities to learn what knowledge BERT actually uses at inference time.

Subfields: Artificial Intelligence, Economics and Econometrics, Management Science and Operations Research, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Management Science and Operations Research

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “Another recent line of work has shown that pretrained models such as BERT are redundant in their capacity, allowing for significant sparsification without much degradation in end metrics [45, 46, 47].”
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020, background): “For example, a recent line of work has shown that pre-trained models such as BERT are redundant in their capacity, allowing for signiﬁcant sparsiﬁcation without much degradation in end metrics (Chen et al., 2020; Prasanna et al., 2020; Desai et al., 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “For BERT, Prasanna et al. (2020) go in the opposite direction: pruning on the basis of importance scores, and interpreting the remaining "good" subnetwork.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “…(Houlsby et al., 2019; Stickland and Murray, 2019), knowledge distillation (Sanh et al., 2019; Jiao et al., 2020), neural architecture search (Guo et al., 2019; Wang et al., 2020a), pruning (Sanh et al., 2020; Prasanna et al., 2020), and better pre-training (Liu et al., 2019; Clark et al., 2020).”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, neutral): “Concurrent with our research, Prasanna et al. [24] also study the lottery ticket hypothesis for BERT models.”
Probing Toxic Content in Large Pre-Trained Language Models (2021, background): “However, we are still investigating the reasons behind this success with the experimental setup side (Rogers et al., 2020; Prasanna et al., 2020).”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “Most of the previous pruning work in NLP has focused on unstructured pruning [36, 7, 50, 31, 57], rather than structured pruning [27, 11].”
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021, methodology): “, 2019) showed that Transformers are remarkably robust to pruning (Gordon et al., 2020; Prasanna et al., 2020; Chen et al., 2020; Michel et al., 2019).”
Transforming Wikipedia Into Augmented Data for Query-Focused Summarization (2019, methodology): “Note that we do not apply one-step method to BERT because it generates the same general subnetwork for all tasks [32]–[34], which is not our focus.”
Playing Lottery Tickets with Vision and Language (2021, background): “The properties of LTH are widely studied for image classification [44, 15, 18, 58, 68, 74, 47], and recently, the observations of LTH have also been evidenced across other fields, such as natural language processing [19, 76, 54, 6, 9], object detection [21], generative adversarial networks [10, 31, ”
A Practical Survey on Faster and Lighter Transformers (2021, methodology): “[36] successfully verified this hypothesis on BERT, even noticing that BERT worst subnetworks remain highly trainable.”
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency (2022, methodology): “The current Transformer-based language models are heavily overparametrized, which explains why it is possible to prune these models by up to 30-40% (Gordon et al., 2020; Sanh et al., 2020; Prasanna et al., 2020; Chen et al., 2020, inter alia) without a significant drop in performance.”
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020, result): “This observation is similar to the results of evaluating the lottery ticket hypothesis on BERT (Prasanna et al., 2020; Chen et al., 2020): a number of subnetworks exist in BERT achieving similar task performance.”