Learning both Weights and Connections for Efficient Neural Network
By Song Han, Jeff Pool, John Tran, William (Bill) J. Dally. 2015
7,826 citations recorded by the source corpus.
Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections. Our method prunes redundant connections using a three-step method. First, we train the network to learn which connections are important. Next, we prune the unimportant connections. Finally, we retrain the network to fine tune the weights of the remaining connections. On the ImageNet dataset, our method reduced the number of parameters of AlexNet by a factor of 9x, from 61 million to 6.7 million, without incurring accuracy loss. Similar experiments with VGG-16 found that the number of parameters can be reduced by 13x, from 138 million to 10.3 million, again with no loss of accuracy.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Quoc V. Le
- Dacheng Tao
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Alan L. Yuille
In Efficiency & Compression
Citations in the corpus
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “…with reduced precision support and (2) reduce overall memory footprint (Han et al., 2016; Dong et al., 2019 Effect of MT Model Size weights to zero to (1) remove operations and (2) reduce the memory footprint when models are stored in sparse matrix formats (LeCun et al., 1990; Han et al., 2015).”
- TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “, 2014), weights pruning (Han et al., 2015), and knowledge distillation (KD) (Romero et al.”
- The State of Sparsity in Deep Neural Networks (2019, methodology): “It has been shown empirically that deep neural networks can tolerate high levels of sparsity (Han et al., 2015; Narang et al., 2017; Ullrich et al., 2017), and this property has been leveraged to significantly reduce the cost associated with the deployment of deep neural networks, and to enable the…”
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “[36] Song Han, Jeff Pool, John Tran, and William J Dally.”
- MegaBlocks: Efficient Sparse Training with Mixture-of-Experts (2022, background): “Exploiting sparsity in the weights, activations and input data of deep neural networks (DNNs) is an effective technique for reducing the amount of computation that is needed to achieve a given model quality (Han et al., 2015; Gale et al., 2019).”
- Cascade R-CNN: High Quality Object Detection and Instance Segmentation (2019, methodology): “To reduce parameters, [23] was used to prune less important connections.”
- Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters (2018, background): “A related but simpler approach was proposed in Han et al. (2015), where small weights are truncated to zero, alternated with re-training.”
- On the Predictability of Pruning Across Scales (2020, methodology): “For decades, neural network pruning—eliminating unwanted parts of a network—has been a popular approach for reducing network sizes or the computational demands of inference (LeCun et al., 1990; Reed, 1993; Han et al., 2015).”
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, methodology): “more parameters than needed in inference (Buciluǎ et al., 2006; Hinton et al., 2015; Urban et al., 2017; Karnin, 1990; Reed, 1993; Augasta and Kathirvalavakumar, 2013; Liu et al., 2014; Han et al., 2015; Molchanov et al., 2017).”
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from different aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
- Cascade R-CNN: Delving Into High Quality Object Detection (2017, methodology): “To reduce parameters, we used [15] to prune less important connections.”
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “As a result, it is often a challenge to run DNNs on target low-power devices, and substantial research efforts are invested in speeding up DNNs at run-time on both generalpurpose (Vanhoucke et al., 2011; Gong et al., 2014; Romero et al., 2014; Han et al., 2015) and specialized computer hardware (Far”
- Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, methodology): “Larger weights play a more important role than smaller weights (Han et al., 2015), but there are fewer of these large weights.”
- Are Sixteen Heads Really Better than One? (2019, background): “orthogonal approaches: fine-grained “weight-by-weight” pruning (Han et al., 2015) and structured pruning (Anwar et al.”
- MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018, methodology): “methods of network pruning [12, 13, 14, 15, 16, 17] and connectivity learning [18, 19].”
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, methodology): “Techniques for eliminating unnecessary weights from neural networks (pruning) (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015; Li et al., 2016) can reduce parameter-counts by more than 90% while maintaining accuracy.”