Compressing Neural Networks with the Hashing Trick

By James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, Yixin Chen. 2015

1,240 citations recorded by the source corpus.

As deep nets are increasingly used in applications suited for mobile devices, a fundamental dilemma becomes apparent: the trend in deep learning is to grow models to absorb ever-increasing data set sizes; however mobile devices are designed with very little memory and cannot store such large models. We present a novel network architecture, HashedNets, that exploits inherent redundancy in neural networks to achieve drastic reductions in model sizes. HashedNets uses a low-cost hash function to randomly group connection weights into hash buckets, and all connections within the same hash bucket share a single parameter value. These parameters are tuned to adjust to the HashedNets weight sharing architecture with standard backprop during training. Our hashing procedure introduces no additional memory overhead, and we demonstrate on several benchmark data sets that HashedNets shrink the storage requirements of neural networks substantially while mostly preserving generalization performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Rethinking the Inception Architecture for Computer Vision (2015, background): “It is certainly possible to mitigate parts of these issues by applying specialized solutions to target memory use [2], [15] or by optimizing the execution of certain operations via computational tricks [10].”
Sequence-Level Knowledge Distillation (2016, background): “There has also been work on reducing the memory footprint of models through quantization of weights (Chen et al., 2015; Han et al., 2016; Courbariaux et al., 2016) as a postprocessing step.”
Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters (2018, background): “A related but simpler approach was proposed in Han et al. (2015), where small weights are truncated to zero, alternated with re-training.”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, background): “Weight tying, e.g. by random hashing of weights into buckets (Chen et al., 2015), is equivalent to subspace training where the subspace is restricted to lie along the equidistant “diagonals” between any axes that are tied together.”
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017, methodology): “We then describe the MobileNet network structure and conclude with descriptions of the two model shrinking hyper-parameters width multiplier and resolution multiplier.”
Learning both Weights and Connections for Efficient Neural Network (2015, methodology): “HashedNets [20] is a recent technique to reduce model sizes by using a hash function to randomly group connection weights into hash buckets, so that all connections within the same hash bucket share a single parameter value.”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, methodology): “HashedNets(Chen et al., 2015) reduce model sizes by using a hash function to randomly group connection weights, so that all connections within the same hash bucket share a single parameter value.”
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, background): “Furthermore, our results carry to other variants of dropout as well (such as drop-connect [29], multiplicative Gaussian noise [13], hashed neural networks [30], etc.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, methodology): “HashedNets [23] uses a hash function to reduce model size by randomly grouping the weights, such that connections in a hash bucket use a single parameter value.”
Quantized Convolutional Neural Networks for Mobile Devices (2015, background): “These approaches mostly focus on compressing parameters in fully-connected layers [11, 2], and none of them can provide acceleration for the test-phase computation.”
Fixed Point Quantization of Deep Convolutional Networks (2015, methodology): “(2014), (b) compressing neural networks using hashing (Chen et al., 2015), and (c) combining pruning and quantization during training to reduce the model size without affecting the accuracy (Han et al.”
Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, methodology): “In HashedNets [143], the neural network connection weights are randomly grouped into hash buckets using a hash function.”
Recent advances in convolutional neural networks (2015, methodology): “t al . [186] merge lters with similar output activations into one. Designing a proper hashing technique to accelerate the training of CNNs or save memory space also an interesting problem. HashedNets [187] is a recent technique to reduce model sizes by using a hash function to group connection weigh”
RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks (2018, background): “The compression of deep networks has also been studied in [1, 10, 11], SqueezeNet [14], etc., where the low-rank factorization of ﬁlters can be utilized.”
Compression of fully-connected layer in neural network by Kronecker product (2015, methodology): “In addition to the low rank methods, hashing methods have also been used to reduce the number of parameters [26]–[28], and distillation offers another way of compressing neural networks [29].”
Compressing Convolutional Neural Networks in the Frequency Domain (2015, methodology): “[6], we achieve smaller models by randomly forcing weights throughout the network to share identical values.”
Fast ConvNets Using Group-Wise Brain Damage (2015, methodology): “Our work focuses on the task of speeding up convolutional layers (as they represent the speed bottleneck) and is therefore complimentary to approaches that focus on the reduction of size/memory footprint of fully-connected layers [11, 18, 38, 42, 46].”
Deep Fried Convnets (2014, methodology): “We also note that any of the techniques from [8, 6] could be applied to the final layer of a deep fried network to further reduce memory consumption at test time.”