An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections

By Yu Cheng, Felix X. Yu, Rogério Schmidt Feris, Sanjiv Kumar, Alok N. Choudhary, Shih-Fu Chang. 2015

341 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent advances in convolutional neural networks (2015, neutral): “[168] explore the redundancy in the parametrization of fully-connected layers by imposing the circulant structure on the weight matrix to speed up the computation, and further allow the use of FFT for faster computation.”
Compression of fully-connected layer in neural network by Kronecker product (2015, methodology): “The Circulant method [11] replaces the conventional linear projection in fully-connected layers with the circulant projection.”
NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, methodology): “[2] exploited properties of structured matrices and used circulant matrices to represent FC layers, reducing storage cost.”
Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications (2015, methodology): “More recently, various methods based on vector quantization (Gong et al., 2014), hashing techniques (Chen et al., 2015), circulant projection (Cheng et al., 2015), and tensor train decomposition (Novikov et al., 2015) were proposed and showed better compression capability than SVD.”
Orthogonal Random Features (2016, methodology): “Structured matrices have been used for speeding up dimensionality reduction [1], binary embedding [27], deep neural networks [5] and kernel approximation [13, 28, 7].”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “Additionally, DFTs have been used to approximate dense, linear layers in neural networks to reduce the parameter count and computational complexity (Cheng et al., 2015; Moczulski et al., 2015; Sindhwani et al., 2015).”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, background): “The need to deploy top-performing deep neural network models on mobile devices motivates techniques that can effectively reduce the storage and computational costs of such networks, including knowledge distillation [19, 40, 4], low-rank factorization [25, 47, 41], filter pruning [30, 36, 32, 57], qu”
Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, methodology): “Methods for compressing and accelerating convolutional networks include knowledge distillation [12, 29], low-rank-factorization [14, 39, 33], pruning and quantization [10, 27], structured matrices [6, 35, 9], and dynamic capacity networks [2].”
Online Ensemble Model Compression Using Knowledge Distillation (2020, methodology): “Parameter Pruning [15,37,6,14], Low Rank Factorization [8,29,36], Transferred Convolutional Filters [9,33,25,11] and Knowledge Distillation methods ar X iv :2 01 1.”
Binary Neural Networks: A Survey (2020, background): “Many prior studies have proven that there usually exists large redundancy in the deep structure [24, 25, 26, 27].”
Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations (2019, methodology): “, 7]→ [0, 4, 2, 6, 1, 5, 3, 7]).”
Dynamic Recursive Neural Network (2019, methodology): “Related techniques focus on many fields like distillation [3, 15], filter pruning [5, 31], low-rank fac torization [21], quantization [12], compression with struc tured matrices [4, 35] and network binarization [32].”
Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, methodology): “Compression using structured matrices translates into inference speed-up, but only for matrices of size 2048 × 2048 and larger (Thomas et al., 2018) on CPUs or when using specialized hardware (Cheng et al., 2015; Sindhwani et al., 2015).”
Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype (2018, background): “(Cheng et al., 2015; Sindhwani et al., 2015) replaced matrix multiplications with structured operations that require less memory.”
Multiscale Hierarchical Convolutional Networks (2017, methodology): “A parameter redundancy has also been observed in the ﬁ-nal fully connected layers used by number of neural network architectures, which contain most of the CNN parameters (Cheng et al., 2015; Lu et al., 2016).”
Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds (2018, methodology): “[12] Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang.”
Training Sparse Neural Networks (2016, methodology): “Cheng et al.[4] make use of circulant matrices to re-paramaterize fully connected layers.”
Risk Prediction with Electronic Health Records: A Deep Learning Approach (2016, background): “One potential future work could be exploit parameters reducing framework [38, 39] to further light the CNN model to prevent over-fitting.”