Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds

By Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, Daniela Rus. 2018

86 citations recorded by the source corpus.

We present an efficient coresets-based neural network compression algorithm that sparsifies the parameters of a trained fully-connected neural network in a manner that provably approximates the network's output. Our approach is based on an importance sampling scheme that judiciously defines a sampling distribution over the neural network parameters, and as a result, retains parameters of high importance while discarding redundant ones. We leverage a novel, empirical notion of sensitivity and extend traditional coreset constructions to the application of compressing parameters. Our theoretical analysis establishes guarantees on the size and accuracy of the resulting compressed network and gives rise to generalization bounds that may provide new insights into the generalization properties of neural networks. We demonstrate the practical effectiveness of our algorithm on a variety of neural network configurations and real-world data sets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data (2018, background): “[3, 23, 21] proved spectrally-normalized margin-based generalization bounds, [9, 23] derived bounds from a PAC-Bayes approach, and [1, 33, 4] derived bounds from the compression point of view.”
Provable Filter Pruning for Efficient Neural Networks (2019, methodology): “To bound the variance, we use an approach inspired by Baykal et al. (2019a) where the main idea is to use the notion of empirical sensitivity to establish that a particular useful inequality holds with high probability over the randomness of the input point x ∼ D .”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “Coreset-Based Compression (CBC) [61] adopts ﬁlter pruning to pre-process ﬁlters for coreset-based compression [226].”
Core‐sets: An updated survey (2019, methodology): “Similarly, we can compress the network neuron by neuron via sensitivities of edges as was suggested in Baykal, Liebenwein, Gilitschenski, Feldman, and Rus (2019) and Sener and Savarese (2017).”
Layer-adaptive Sparsity for the Magnitude-based Pruning (2020, background): “…target weight to use as an importance score, e.g., Hessian-based approximations (LeCun et al., 1989; Hassibi & Stork, 1993; Dong et al., 2017), coreset-based estimates (Baykal et al., 2019; Mussay et al., 2020), convex optimization (Aghasi et al., 2017), and operator distortion (Park et al., 2020).”
Data-Independent Structured Pruning of Neural Networks via Coresets (2020, methodology): “Similar to our work, the approaches in [4] and [35] use coresets for model compression.”
On Measuring Excess Capacity in Neural Networks (2022, background): “Recent works range from relating generalization to weight norms [3, 14, 26, 37, 38], measures of the distance to initializations [34], implicit regularization induced by the optimization algorithm [8, 41], or model compression [2, 6, 42].”
Data-Independent Neural Pruning via Coresets (2019, methodology): “Similar to our approach, Baykal et al. (2018) used coresets for model compression.”