The Lottery Ticket Hypothesis for Pre-trained BERT Networks

By Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, Michael Carbin. 2020

422 citations recorded by the source corpus.

In natural language processing (NLP), enormous pre-trained models like BERT have become the standard starting point for training on a range of downstream tasks, and similar trends are emerging in other areas of deep learning. In parallel, work on the lottery ticket hypothesis has shown that models for NLP and computer vision contain smaller matching subnetworks capable of training in isolation to full accuracy and transferring to other tasks. In this work, we combine these observations to assess whether such trainable, transferrable subnetworks exist in pre-trained BERT models. For a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity. We find these subnetworks at (pre-trained) initialization, a deviation from prior NLP research where they emerge only after some amount of training. Subnetworks found on the masked language modeling task (the same task used to pre-train the model) transfer universally; those found on other tasks transfer in a limited fashion if at all. As large-scale pre-training becomes an increasingly central paradigm in deep learning, our results demonstrate that the main lottery ticket observations remain relevant in this context. Codes available at this https URL.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, background): “The remarkable success of those works have sparked interest the lottery-ticket hypothesis (Frankle and Carbin, 2019; Chen et al., 2020; Prasanna et al., 2020): the conjecture that large models are needed in pretraining only to induce (in high probability) the existing of sub-networks initialized…”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “Another recent line of work has shown that pretrained models such as BERT are redundant in their capacity, allowing for significant sparsification without much degradation in end metrics [45, 46, 47].”
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020, background): “For example, a recent line of work has shown that pre-trained models such as BERT are redundant in their capacity, allowing for significant sparsification without much degradation in end metrics (Chen et al., 2020; Prasanna et al., 2020; Desai et al., 2019).”
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020, result): “This observation is similar to the results of evaluating the lottery ticket hypothesis on BERT (Prasanna et al., 2020; Chen et al., 2020): a number of subnetworks exist in BERT achieving similar task performance.”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, background): “Unstructured pruning (Frankle & Carbin, 2018; Li et al., 2020; Chen et al., 2020; Sanh et al., 2020) prunes individual neurons instead of structured blocks.”
BERT Busters: Outlier Dimensions that Disrupt Transformers (2021, background): “Some of the recent ﬁndings are that the lottery ticket hypothesis (Frankle and Carbin, 2019) holds for BERT: its largest weights do form subnetworks that can be retrained alone to reach the performance close to that of the full model (Prasanna et al., 2020; Chen et al., 2020; Gordon et al., 2020).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “One explanation for the effectiveness of these approaches is the Lottery Ticket Hypothesis [16, 105, 140], which suggests that a subnetwork within the model does most of the heavy lifting, and the remaining weights can all be pruned away.”
Playing Lottery Tickets with Vision and Language (2021, methodology): “To assess the importance of the initialization, we consider two different initializations with the learned mask unchanged: (i) using pre-trained BERT weights θ0 as initialization, and (ii) shuffling the UNITER pre-trained weights within each layer to obtain a new initialization θ′′ 0 as used in [6].”
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, result): “Pruning-based approaches [118], [119] aim to reduce the size of the model by removing a number of unimportant weights, while achieving similar performance to the original model.”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, methodology): “Most of the previous pruning work in NLP has focused on unstructured pruning [36, 7, 50, 31, 57], rather than structured pruning [27, 11].”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “Some works ([121], [187], [195], [196]) study the transfer-ability of sparsity masks on datasets.”
A Simple and Effective Pruning Approach for Large Language Models (2023, neutral): “A quick review of existing pruning methods reveals a possible reason: they typically require retraining [3, 25], training from scratch [8, 32, 44, 54] or even an extensive iterative process [23, 37, 77].”