Data-Independent Neural Pruning via Coresets

By Ben Mussay, Margarita Osadchy, Vladimir Braverman, Samson Zhou, Dan Feldman. 2019

67 citations recorded by the source corpus.

Previous work showed empirically that large neural networks can be significantly reduced in size while preserving their accuracy. Model compression became a central research topic, as it is crucial for deployment of neural networks on devices with limited computational and memory resources. The majority of the compression methods are based on heuristics and offer no worst-case guarantees on the trade-off between the compression rate and the approximation error for an arbitrarily new sample. We propose the first efficient, data-independent neural pruning algorithm with a provable trade-off between its compression rate and the approximation error for any future test sample. Our method is based on the coreset framework, which finds a small weighted subset of points that provably approximates the original inputs. Specifically, we approximate the output of a layer of neurons by a coreset of neurons in the previous layer and discard the rest. We apply this framework in a layer-by-layer fashion from the top to the bottom. Unlike previous works, our coreset is data independent, meaning that it provably guarantees the accuracy of the function for any input $x\in \mathbb{R}^d$, including an adversarial one. We demonstrate the effectiveness of our method on popular network architectures. In particular, our coresets yield 90\% compression of the LeNet-300-100 architecture on MNIST while improving the accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Some works prune neurons/channels individually based on some importance score [He et al., 2014, Li et al., 2017, Liebenwein et al., 2020, Mussay et al., 2020, 2021, Molchanov et al., 2017, Srinivas and Babu, 2015].”
Layer-adaptive Sparsity for the Magnitude-based Pruning (2020, background): “…target weight to use as an importance score, e.g., Hessian-based approximations (LeCun et al., 1989; Hassibi & Stork, 1993; Dong et al., 2017), coreset-based estimates (Baykal et al., 2019; Mussay et al., 2020), convex optimization (Aghasi et al., 2017), and operator distortion (Park et al., 2020).”
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot (2020, methodology): “Therefore, there are only a few data-independent pruning methods [34], and it is not clear whether these methods can get similar (or even better) performance than the popular data-dependent pruning methods.”
Towards Total Recall in Industrial Anomaly Detection (2021, methodology): “More recently, coreset-based methods have also found their way into Deep Learning approaches, e.g for network pruning [34], active learning [48] and increasing effective data coverage of mini-batches for improved GAN training [49] or representation learning [41].”
Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough (2020, methodology): “Recently, a line of works on network pruning with theoretical guarantees have emerged, including sensitivity-based methods (Baykal et al., 2019b; Liebenwein et al., 2020), coreset-based methods (Baykal et al., 2019a; Mussay et al., 2020), greedy forward selection (Ye et al., 2020).”