Pruning algorithms of neural networks — a comparative study

By Gethsiyal Augasta M, Thangairulappan Kathirvalavakumar. 2013

130 citations recorded by the source corpus.

The neural network with optimal architecture speeds up the learning process and generalizes the problem well for further knowledge extraction. As a result researchers have developed various techniques for pruning the neural networks. This paper provides a survey of existing pruning techniques that optimize the architecture of neural networks and discusses their advantages and limitations. Also the paper evaluates the effectiveness of various pruning techniques by comparing the performance of some traditional and recent pruning algorithms based on sensitivity analysis, mutual information and significance on four real datasets namely Iris, Wisconsin breast cancer, Hepatitis Domain and Pima Indian Diabetes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, background): “, 2017; Karnin, 1990; Reed, 1993; Augasta and Kathirvalavakumar, 2013; Liu et al., 2014; Han et al., 2015; Molchanov et al., 2017). Gordon et al. (2020) have demonstrated that overparmeterization can be exploited in finetuning: pruned network perform well in transfer setting.”
Compression of Neural Machine Translation Models via Pruning (2016, background): “However, the high computational complexity of OBD and OBS compare unfavorably to the computational simplicity of the magnitude-based approach, especially for large networks (Augasta and Kathirvalavakumar, 2013).”
A Systematic Review on Model Watermarking for Neural Networks (2020, background): “• Pruning (Augasta and Kathirvalavakumar, 2013; Molchanov et al., 2019): cut some redundant parameters and obtain a new model that looks different from the original model but still has a similarly high prediction accuracy.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “Though pruning sometimes introduce little improvement of the accuracy as they remove some of the local minima [12], the techniques can not perform better than switch to a better architecture [24], e.”