Compressing deep neural networks using a rank-constrained topology

By Preetum Nakkiran, Raziel Alvarez, Rohit Prabhavalkar, Carolina Parada. 2015

81 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Signal Processing

Citations in the corpus

Initialization and Regularization of Factorized Neural Layers (2021, methodology): “For the former, several papers use SI after training (Nakkiran et al., 2015; Yaguchi et al., 2019; Yang et al., 2020), while Ioannou et al.”
Composition of Deep and Spiking Neural Networks for Very Low Bit Rate Speech Coding (2016, background): “The low-rank factorization and Toeplitz matrices are found to enable significant memory reduction with minor or no reduction in performance [75], [76].”
On the compression of recurrent neural networks with an application to LVCSR acoustic modeling for embedded speech recognition (2016, methodology): “coustic models. The methods investigated in our work are most similar to previous work that has examined using SVD to reduce the number of parameters in the network in the context of feedforward DNNs [9, 10, 22]. As we describe in Section 3, our methods can be thought of as an extension of the techn”
Hello Edge: Keyword Spotting on Microcontrollers (2017, methodology): “Furthermore, low-rank approximation techniques are used to compress the DNN model weights achieving similar accuracy with less hardware resources [15, 16].”
Training Keyword Spotting Models on Non-IID Data with Federated Learning (2020, methodology): “The model consisted of an encoderdecoder architecture in which both the encoder and decoder made use of efficiently parameterized SVDF (single value decomposition filter) layers—originally introduced in [26]—to approximate fully-connected layers with low rank approximations.”
Deep Model Compression: Distilling Knowledge from Noisy Teachers (2016, methodology): “Matrix decomposition methods: Denil et al. [5], Sainath et al. [25] and Nakkiran et al. [21] all adopted low-rank decomposition to compress the weights in different layers.”
On-Device Neural Language Model Based Word Prediction (2018, methodology): “Compression through matrix factorization (Sainath et al., 2013; Xue et al., 2013; Nakkiran et al., 2015; Prabhavalkar et al., 2016; Lu et al., 2016) has shown promising results in model compression but has been applied to the tasks of automatic speech recognition.”
NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, methodology): “Within the structured pruning family, there are approaches based on the singular value decomposition (SVD) (Xue et al., 2013; Nakkiran et al., 2015; Prabhavalkar et al., 2016), where a new set of ﬁlters are obtained by projecting the original weights onto a lower dimensional space.”
Personalized speech recognition on mobile devices (2016, methodology): “SVD has elsewhere been shown to be effective for speech processing tasks [4, 6, 7] as have structured transforms [8] and low-rank matrix factorizations [9].”