Provable Filter Pruning for Efficient Neural Networks

By Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, Daniela Rus. 2019

167 citations recorded by the source corpus.

We present a provable, sampling-based approach for generating compact Convolutional Neural Networks (CNNs) by identifying and removing redundant filters from an over-parameterized network. Our algorithm uses a small batch of input data points to assign a saliency score to each filter and constructs an importance sampling distribution where filters that highly affect the output are sampled with correspondingly high probability. In contrast to existing filter pruning approaches, our method is simultaneously data-informed, exhibits provable guarantees on the size and performance of the pruned network, and is widely applicable to varying network architectures and data sets. Our analytical bounds bridge the notions of compressibility and importance of network structures, which gives rise to a fully-automated procedure for identifying and preserving filters in layers that are essential to the network's performance. Our experimental evaluations on popular architectures and data sets show that our algorithm consistently generates sparser and more efficient models than those constructed by existing filter pruning approaches.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Unless otherwise specified, we use the per-layer budget selection method described in Section 5.2 for all the layerwise pruning methods, except for LAYERSAMPLING for which we use its own budget selection strategy provided in [Liebenwein et al., 2020].”
Neural Pruning via Growing Regularization (2020, background): “(3) Many methods work on the weight importance criterion problem, including some very recent ones (ProvableFP Liebenwein et al. (2020), LFPC He et al. (2020)).”
Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough (2020, background): “Recently, a line of works on network pruning with theoretical guarantees have emerged, including sensitivity-based methods (Baykal et al., 2019b; Liebenwein et al., 2020), coreset-based methods (Baykal et al., 2019a; Mussay et al., 2020), greedy forward selection (Ye et al., 2020).”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “3 All Layer NISP [69], DCP [70], PFP [71], DLRFC [72]”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “) from different perspectives, including magnitude ([10, 139, 140, 141]), norm ([69, 86]), saliency and/or sensitivity ([85, 142, 143]), loss change ([13, 50, 71, 142, 144]).”
Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection (2020, unknown): “However, these observations are different from the recent findings of Liu et al. (2019b), whose empirical results suggest that training a large, over-parameterized network is often not necessary for obtaining an efficient small network and finetuning the pruned subnetwork is no better than retrainin”
Data-Independent Structured Pruning of Neural Networks via Coresets (2020, methodology): “Similar to our work, the approaches in [4] and [35] use coresets for model compression.”