Compressing Convolutional Neural Networks in the Frequency Domain
By James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, Yixin Chen. 2015
155 citations recorded by the source corpus.
Convolutional neural networks (CNN) are increasingly used in many areas of computer vision. They are particularly attractive because of their ability to "absorb" great quantities of labeled data through millions of parameters. However, as model sizes increase, so do the storage and memory requirements of the classifiers, hindering many applications such as image and speech recognition on mobile phones and other devices. In this paper, we present a novel net- work architecture, Frequency-Sensitive Hashed Nets (FreshNets), which exploits inherent redundancy in both convolutional layers and fully-connected layers of a deep learning model, leading to dramatic savings in memory and storage consumption. Based on the key observation that the weights of learned convolutional filters are typically smooth and low-frequency, we first convert filter weights to the frequency domain with a discrete cosine transform (DCT) and use a low-cost hash function to randomly group frequency parameters into hash buckets. All parameters assigned the same hash bucket share a single value learned with standard back-propagation. To further reduce model size, we allocate fewer hash buckets to high-frequency components, which are generally less important. We evaluate FreshNets on eight data sets, and show that it leads to better compressed performance than several relevant baselines.
Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
In Efficiency & Compression
Citations in the corpus
- Quantized Convolutional Neural Networks for Mobile Devices (2015, background): “, if the inner products between each sub-vectory(m) and all the sub-codewords in the m-th sub-codebook have been computed in advance. Quantization-based approaches have been explored in several works [10, 3, 2, 11]. These approaches mostly focus on compressing parameters in fully-connected layers [1”
- Machine Learning Models that Remember Too Much (2017, background): “Several papers investigated how to compress models [13, 15, 29].”
- Compression of fully-connected layer in neural network by Kronecker product (2015, methodology): “In addition to the low rank methods, hashing methods have also been used to reduce the number of parameters [26]–[28], and distillation offers another way of compressing neural networks [29].”
- Improved Techniques for Training Adaptive Deep Networks (2019, methodology): “The other line of research explores methods to prune [8, 23, 11] or quantize [4, 8, 17] neural network weights.”
- Cognitive computation and communication: A complement solution to cloud for IoT (2016, unknown): “The shared weight approach is also found in HashedNets [52].”
- FcaNet: Frequency Channel Attention Networks (2020, methodology): “There are also some applications in the model compression and pruning tasks like [6, 27, 38].”
- Communication-Efficient Edge AI: Algorithms and Systems (2020, methodology): “The subsequent work [63] extended it to CNNs, which is achieved by first converting filter weights to the frequency domain and then grouping the corresponding frequency parameters into hash buckets using a low-cost hash function.”
- Frequency-Domain Dynamic Pruning for Convolutional Neural Networks (2018, methodology): “[14] extended the quantization method into frequency domain and used a hash function to randomly group frequency parameters into hash buckets and all parameters assigned to”
- Focal Frequency Loss for Image Reconstruction and Synthesis (2020, background): “Besides, several studies have incorporated frequency analysis with network compression [16, 66, 68, 6, 17] and feature reduction [35, 61] to accelerate the training and inference of networks.”