Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets

By Tara N. Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, Bhuvana Ramabhadran. 2013

715 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Signal Processing

Citations in the corpus

Generating Sequences With Recurrent Neural Networks (2013, background): “…a further diﬃculty is the high computational cost of evaluating all the exponentials during training (although several methods have been to devised make training large softmax layers more eﬃcient, including tree-based models [25, 23], low rank approximations [27] and stochastic derivatives [26]).”
Data-Efficient Structured Pruning via Submodular Optimization (2022, neutral): “, 2015), while another set of methods use low-rank approximation or tensor factorization (Denil et al., 2013; Sainath et al., 2013; Jaderberg et al., 2014; Denton et al., 2014; Lebedev et al., 2015; Kim et al., 2015; Zhang et al., 2016; Su et al., 2018).”
LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “Some prior works even explicitly impose the low-rank constraint when training the original neural network, such as [17, 30, 35, 38, 39], however, to the best of our knowledge, none of these works considers low-rank update for adaptation to downstream tasks.”
Do Deep Nets Really Need to be Deep? (2013, background): “If a shallow net with the same number of parameters as a deep net can learn to mimic a deep net with high fidelity, then it is clear that the function learned by that deep net does not really have to be deep.”
Collaborative Deep Learning for Recommender Systems (2014, methodology): “[24] uses low-rank matrix factorization in the last weight layer of a deep network to significantly reduce the number of model parameters and speed up training, but it is for classification instead of recommendation tasks.”
Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks (2020, background): “Work on standard supervised learning has alternatively studied low-dimensional structures via rank (Goldblum et al., 2019b; Sainath et al., 2013).”
Recent advances in convolutional neural networks (2015, methodology): “To this end, [78] applies the low-rank matrix factorization to the final weight layer in a”
Deep Learning: Methods and Applications (2014, methodology): “In [310] and [397], the DNN with high-dimensional output layer was trained first.”
Deep Fried Convnets (2014, methodology): “In contrast, the methods advanced in [33] and this paper apply both at train and test time.”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, methodology): “Finally, low-rank factorizations and sketches of the gradients when viewed as matrices were proposed in [35, 36, 37, 38, 12]; arguably most of these methods (with the exception of [12]) aimed to address the high flops required when training low-rank models.”
Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition (2014, background): “For consistency with the LSTM architectures, some of the networks have a low-rank projection layer [17].”
Diversity Networks (2015, methodology): “Sainath et al. (2013) reduce the number of parameters by analyzing the weight matrices, and applying low-rank factorization to the final weight layer.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “…architectures : One line of work has focused on optimizing the NN model architecture in terms of its micro-architecture [100, 110, 125, 163, 164, 205, 245, 272] (e.g., kernel types such as depth-wise convolution or low-rank factorization) as well as its macro-architecture [99, 100, 103, 109,…”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, methodology): “The need to deploy top-performing deep neural network models on mobile devices motivates techniques that can effectively reduce the storage and computational costs of such networks, including knowledge distillation [19, 40, 4], low-rank factorization [25, 47, 41], filter pruning [30, 36, 32, 57], qu”
Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, methodology): “Methods for compressing and accelerating convolutional networks include knowledge distillation [12, 29], low-rank-factorization [14, 39, 33], pruning and quantization [10, 27], structured matrices [6, 35, 9], and dynamic capacity networks [2].”
Moonshine: Distilling with Cheap Convolutions (2017, methodology): “Equally, there are many methods for low rank approximations that could be applicable (Sainath et al., 2013; Jaderberg et al., 2014; Garipov et al., 2016).”
Lower Frame Rate Neural Network Acoustic Models (2016, background): “Again, to reduce the number of parameters, the softmax layer is factored into two with an intermediate 512-node linear low-rank layer [14].”