Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

By Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang. 2016

967 citations recorded by the source corpus.

State-of-the-art neural networks are getting deeper and wider. While their performance increases with the increasing number of layers and neurons, it is crucial to design an efficient deep architecture in order to reduce computational and memory costs. Designing an efficient neural network, however, is labor intensive requiring many experiments, and fine-tunings. In this paper, we introduce network trimming which iteratively optimizes the network by pruning unimportant neurons based on analysis of their outputs on a large dataset. Our algorithm is inspired by an observation that the outputs of a significant portion of neurons in a large network are mostly zero, regardless of what inputs the network received. These zero activation neurons are redundant, and can be removed without affecting the overall accuracy of the network. After pruning the zero activation neurons, we retrain the network using the weights before pruning as initialization. We alternate the pruning and retraining to further reduce zero activations in a network. Our experiments on the LeNet and VGG-16 show that we can achieve high compression ratio of parameters without losing or even achieving higher accuracy than the original network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, background): “, L1, L2 norms [27, 5, 20, 26], firstorder [29, 3] or second-order [19, 7, 2] gradients, average percentage of zero [26, 11], singular values [32, 28].”
You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization (2018, background): “For neuron level pruning, several works prune the neurons directly by evaluating the importance of neuron based on speciﬁc criteria (Hu et al. (2016) (Zoph et al. (2018)), early stopping with performance prediction (Zhong et al. (2018)), progressive search (Liu et al. (2018a)) or weight sharing…”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, background): “…et al., 2018; Luo et al., 2017; He et al., 2017; Molchanov et al., 2017; He et al., 2018b; Yang et al., 2018a; Jiang et al., 2018; He et al., 2018a; Hu et al., 2016; Liu et al., 2017; Li et al., 2017; Lin et al., 2018; Wen et al., 2016) given its ﬁne-grained control capability over the constraints.”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “Channel pruning [40, 24, 1, 38] removes the redundant channels from feature maps.”
NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, background): “Activations are often used to determine which units are more important for a given task in problems such as CL (Jung et al., 2020a; Golkar et al., 2019), network pruning (Kurtz et al., 2020; Hu et al., 2016; Rhu et al., 2018), and model interpretation (Erhan et al., 2009; Zeiler & Fergus, 2014).”
Continual Learning via Neural Pruning (2019, methodology): “For accessibility, we use a simple neuron/filter based sparsification scheme which can be thought of as a single iteration variation of [7].”
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression (2017, methodology): “APoZ [14] aims to reduce parameter numbers, but its performance is limited.”
Rethinking the Value of Network Pruning (2018, methodology): “, 2017) and average percentage of zeros in the output (Hu et al., 2016).”
Data-Driven Sparse Structure Selection for Deep Neural Networks (2017, background): “To tackle this issue, several works pruned the neurons directly [15, 23, 28] by evaluating neuron importance on specific criteria.”
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning (2016, background): “[12] proposed removing filters that frequently generate zero outputs after the ReLU layer in the validation set.”
Extreme Network Compression via Filter Group Approximation (2018, background): “the magnitude of weights [6, 21], sparsity of activation [11], Taylor expansion [24], etc.”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “Some existing methods (Babaeizadeh et al., 2016; Dubey et al., 2018) have explored structured pruning based on activation statistics of neuron/filter output, e.g. percentage of zero activations (Hu et al., 2016) and activation mean (Molchanov et al., 2017).”
A systematic review on overfitting control in shallow and deep neural networks (2021, methodology): “Hu et al. (2016) introduces a network trimming method for deep networks that iteratively prunes unimportant neurons based on output analysis.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “average percentage of zeros (APoZ) [100] was introduced to judge if one output activation map is contributing to the result.”