Coreset-Based Neural Network Compression

By Abhimanyu Dubey, Moitreya Chatterjee, Narendra Ahuja. 2018

87 citations recorded by the source corpus.

We propose a novel Convolutional Neural Network (CNN) compression algorithm based on coreset representations of filters. We exploit the redundancies extant in the space of CNN weights and neuronal activations (across samples) in order to obtain compression. Our method requires no retraining, is easy to implement, and obtains state-of-the-art compression performance across a wide variety of CNN architectures. Coupled with quantization and Huffman coding, we create networks that provide AlexNet-like accuracy, with a memory footprint that is 832\(\times \) smaller than the original AlexNet, while also introducing significant reductions in inference time as well. Additionally these compressed networks when fine-tuned, successfully generalize to other domains as well.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yi Yang
2. Luc Van Gool
3. Dacheng Tao
4. Xiaogang Wang
5. Alan L. Yuille
6. Qi Tian
7. Jitendra Malik
8. Trevor Darrell

Citations in the corpus

• What is the State of Neural Network Pruning? (2020, background): “…(Louizos et al., 2017; Yang et al., 2017; Han et al., 2015; Kim et al., 2015), memory bandwidth usage (Peng et al., 2018; Kim et al., 2015), or fine-tuning time (Dubey et al., 2018; Yamamoto & Maeno, 2018; Huang & Wang, 2018; He et al., 2018a), and we consider this a limitation of our analysis.”
• Similarity-Preserving Knowledge Distillation (2019, methodology): “Techniques from different categories have also been optimized jointly or combined sequentially to achieve higher compression rates [7, 11, 33].”
• Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks (2019, methodology): “[6] minimizes the reconstruction error of training set sample activations and applies Singular Value Decomposition (SVD) to obtain a decomposition of filters.”
• Variational Convolutional Neural Network Pruning (2019, background): “However, most of these pruning approaches [1, 7, 16] need re-training stage to restore comparable accuracy, which increases computational complexity and is not friendly for operation.”
• A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “Some existing methods (Babaeizadeh et al., 2016; Dubey et al., 2018) have explored structured pruning based on activation statistics of neuron/filter output, e.g. percentage of zero activations (Hu et al., 2016) and activation mean (Molchanov et al., 2017).”
• Data-Independent Neural Pruning via Coresets (2019, methodology): “compress existing models have been proposed (see Gale et al. (2019) for a recent review on network sparsification, and Mozer & Smolensky (1989); Srivastava et al. (2014); Yu et al. (2018); He et al. (2017) for neural pruning). Although a variety of model compression heuristics have been successfully”
• Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration (2018, methodology): “[7] aims to obtain a decomposition by minimizing the reconstruction error of training set sample activation.”
• Data-Independent Structured Pruning of Neural Networks via Coresets (2020, methodology): “In [10], k-means coresets were suggested to compress layers by adding a sparsity constraint.”
• Low-Memory Neural Network Training: A Technical Report (2019, methodology): “More recent techniques based on similar principles include (Dubey et al., 2018, Reagen et al., 2018, Wang et al., 2018, Wen et al., 2016).”
• Model Compression by Entropy Penalized Reparameterization (2019, methodology): “For our ImageNet experiments, we evaluate our method on the ResNet-18 and ResNet-50 (He et al., 2016a) networks, and compare it to Coreset-based compression (Dubey et al., 2018), a recent state of the art compression technique that includes pruning and Deep Compression.”
• Pruning Filter via Geometric Median for Deep Convolutional Neural Networks Acceleration (2018, methodology): “Some filter pruning approaches [16, 17, 13, 18, 5, 23, 29, 27] are data dependent, which means the training data is utilized to determine the pruned filters.”
• Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “Coreset-Based Compression (CBC) [61] adopts filter pruning to pre-process filters for coreset-based compression [226].”