The Lottery Ticket Hypothesis: Training Pruned Neural Networks

By Jonathan Frankle, Michael Carbin. 2018

289 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “Frankle & Carbin (2018) follow a similar procedure, but use the same weight initialization that was used when the sparse weight mask was learned and do not perform the longer training time variant.”
Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, background): “Pruning of networks [53] has also been shown to be a step towards understanding the role of individual neurons in networks.”
Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking (2021, neutral): “To avoid constructing a huge network, we compress the model for the current task by releasing a certain fraction of ne-glectable weights of old tasks (Frankle and Carbin, 2019; Geng et al., 2021).”
Rethinking the Value of Network Pruning (2018, background): “Frankle & Carbin (2018) hypothesizes that certain connections, together with their randomly initialized weights, is particularly effective for training, and a pruning algorithm can help ﬁnd such sub-networks.”
Superposition of many models into one (2019, background): “More surprisingly, Frankle & Carbin (2018); Liu et al. (2018) show the pruning mask acquired after training can be applied to the same network before training with little to no adverse impact on learning.”
Importance Estimation for Neural Network Pruning (2019, background): “Empirically, while larger networks have exhibited better performance, possibly due to the lottery ticket hypothesis [5], they have also been known to be heavily over-parameterized [34].”
Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle (2018, background): “While currently the network architectures leading to state-of-the-art performance in various classification tasks are highly over-parameterized, it has also been observed that a major portion of the network parameters can be pruned without significant deterioration in performance [14]– [16] and it h”
Block Pruning For Faster Transformers (2021, methodology): “Approaches such as magnitude pruning (Han et al., 2015), L0 regularization (Louizos et al., 2018), lottery ticket hypothesis (Frankle and Carbin, 2018), diff pruning (Guo et al., 2020), and movement pruning (Sanh et al., 2020) have demonstrated remarkable reductions in model size.”
Continual Learning with Deep Artificial Neurons (2020, background): “The Lottery Ticket Hypothesis has shown that there often exist smaller, easily optimizable networks embedded in the weight matrices of much larger networks [7], again hinting at the existence of optimal structure for specific tasks.”
Accelerating Sparse Deep Neural Networks (2021, methodology): “Some approaches train models with a ﬁxed sparsity pattern starting from the random initialization of the network, but the pattern is computed based on a fully trained [13] or partially trained [24,25] dense model of the same architecture.”
Structured Pruning of Neural Networks With Budget-Aware Regularization (2018, background): “This can be explained by the following hypothesis: the initial value of a filter’s weights is not guaranteed to allow the learning of a useful feature; thus, a trained network might contain many expendable features [7].”
Born-Again Tree Ensembles (2020, methodology): “Focusing on feed-forward neural networks, Frankle & Carbin (2018) proposed pruning techniques that identify subnetworks which perform close to the original network.”
Edinburgh Research Explorer Understanding Learning Dynamics Of Language Models with SVCCA (unknown): “This is necessary because many dimensions in the representations are noisy, and in fact cancel each other out (Frankle and Carbin, 2018).”