Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions

By Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Narayanan Veeraraghavan, Yingyan Lin. 2018

121 citations recorded by the source corpus.

The current trend of pushing CNNs deeper with convolutions has created a pressing demand to achieve higher compression gains on CNNs where convolutions dominate the computation and parameter amount (e.g., GoogLeNet, ResNet and Wide ResNet). Further, the high energy consumption of convolutions limits its deployment on mobile devices. To this end, we proposed a simple yet effective scheme for compressing convolutions though applying k-means clustering on the weights, compression is achieved through weight-sharing, by only recording $K$ cluster centers and weight assignment indexes. We then introduced a novel spectrally relaxed $k$-means regularization, which tends to make hard assignments of convolutional layer weights to $K$ learned cluster centers during re-training. We additionally propose an improved set of metrics to estimate energy consumption of CNN hardware implementations, whose estimation results are verified to be consistent with previously proposed energy estimation tool extrapolated from actual hardware measurements. We finally evaluated Deep $k$-Means across several CNN models in terms of both compression ratio and energy consumption reduction, observing promising results without incurring accuracy loss. The code is available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “Beyond scalar quantization, vector quantization was widely adopted too in model compression for parameter sharing (Gong et al., 2014; Wu et al., 2018).”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, background): “This includes pruning (in which parts of a model are removed) [27–31], quantization (in which parameters are represented with fewer bits) [32, 33], parameter-sharing (in which the same parameters are used in multiple parts of a model) [34–36], and distilliation (in which a smaller student model is t”
EnergyNet: Energy-Efficient Dynamic Inference (2018, background): “Most of them aim to reduce the model size or the number of 28 computations [3, 4, 5, 6, 7, 8, 9, 10, 11, 12].”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, background): “…et al., 2017; He et al., 2018; Wen et al., 2016; Luo et al., 2017; Liu et al., 2018a), weight factorization (Denton et al., 2014), weight sharing (Wu et al., 2018a), quantization (Hubara et al., 2017), dynamic inference (Wang et al., 2018b; 2019b; Shen et al., 2020), network architecture search…”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, methodology): “[27] proposed deep k-means to enable weight-clustering during re-training.”
Cascaded Projection: End-To-End Network Compression and Acceleration (2019, methodology): “In [52] k-means clustering is used to encourage similarity between features to aid in compression.”
Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference (2020, methodology): “Layer factorization (Tai et al., 2016; Yu et al., 2017), quantization (Wu et al., 2016), model distillation (Wang et al., 2018c) and weight sharing (Wu et al., 2018) have also been respectively found effective.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Note that there also exist methods with data clustering rather than directly quantizing them to lower bitwidth [185], [186].”
NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, methodology): “work pruning. Quantization algorithms compress networks by reducing the number of bits used to represent each weight (Wu et al., 2016; Han et al., 2016a; Rastegari et al., 2016; He & Cheng, 2018; Wu et al., 2018). Knowledge distillation (Hinton et al., 2014) and model compression (Buciluˇa et al”
DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better (2019, background): “The demands of efﬁcient restoration model have recently drawn increasing attentions due to the prevailing need of mobile on-device image enhancement [54, 50, 47].”