Restructuring of deep neural network acoustic models with singular value decomposition

By Jian Xue, Jinyu Li, Yifan Gong. 2013

448 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dong Yu
3. Yoshua Bengio
4. Li Deng
5. Quoc V. Le
6. Jianfeng Gao
7. Dacheng Tao
8. Mohit Bansal

In Efficiency & Compression

1. Song Han
2. Ji Lin
3. Jinyu Li
4. Zhijian Liu
5. Dong Yu
6. Hanrui Wang
7. José Manuel Álvarez
8. Yihui He

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Li Deng
4. Yifan Gong
5. Tara N. Sainath
6. Khe Chai Sim
7. Shinji Watanabe
8. Rohit Prabhavalkar

Citations in the corpus

• Fast R-CNN (2015, background): “Recognizing this fact, we look to truncated SVD, which offers a simple way to compress fully-connected layers [5, 22].”
• Do Deep Nets Really Need to be Deep? (2013, methodology): “ic the exact behaviours of a teach model. Moreover, consider a second training case where the teacher predicts logits [ 10;0;10]. After softmax, these logits yield the same predicted probabilities as [10;20;30], yet clearly the teacher has learned internally to model these two cases very differently”
• Recent advances in convolutional neural networks (2015, methodology): “Similarly, [79] applies singular value decomposition on each layer of a deep CNN to reduce the model size by 71% with less than 1% relative accuracy loss.”
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “Tensor factorization [30, 15, 27, 35] decomposes weights into light-weight pieces, for example [51, 11, 14] proposed to accelerate the fully connected layers with truncated SVD; Jaderberg et al .”
• Deep Learning: Methods and Applications (2014, methodology): “In [310] and [397], the DNN with high-dimensional output layer was trained first.”
• Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting (2018, methodology): “The use of layer decomposition with SVD was also investigated in the context of network compression [20], [21].”
• Compression of fully-connected layer in neural network by Kronecker product (2015, methodology): “Our experiments are done on a private dataset for the moment and may extend to other established Chinese character recognition datasets like HCL2000(Zhang et al. (2009)) and CASIA-HWDB(Liu et al. (2013)).”
• Fast ConvNets Using Group-Wise Brain Damage (2015, methodology): “Our work focuses on the task of speeding up convolutional layers (as they represent the speed bottleneck) and is therefore complimentary to approaches that focus on the reduction of size/memory footprint of fully-connected layers [11, 18, 38, 42, 46].”
• Deep Fried Convnets (2014, methodology): “However, first training a full model, then preforming an SVD of the weight matrices followed by a fine tuning phase preserves much of the performance of the original model [37].”
• Selfless Sequential Learning (2018, background): “SVD decomposition can be applied to reduce the number of effective parameters (Xue et al., 2013).”
• Channel Pruning for Accelerating Very Deep Neural Networks (2017, background): “[51, 10, 12] accelerate fully connected layers with truncated SVD.”
• ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, methodology): “Finally, low-rank factorizations and sketches of the gradients when viewed as matrices were proposed in [35, 36, 37, 38, 12]; arguably most of these methods (with the exception of [12]) aimed to address the high flops required when training low-rank models.”
• Extreme Network Compression via Filter Group Approximation (2018, background): “A series of works [4, 16, 38, 41] have achieved great progress on increasing efficiency of networks by representing the original weights as a form of low-rank.”
• Domain-Adaptive Deep Network Compression (2017, methodology): “In the original paper [21, 7, 38] the compression is always applied on the source domain, and no analysis of its efficiency for domain transfer exists.”
• Object Detection With Deep Learning: A Review (2018, methodology): “The truncated singular value decomposition (SVD) [91] can be utilized to compress large FC layers and to accelerate the testing procedure.”
• ADC: Automated Deep Compression and Acceleration with Reinforcement Learning (2018, background): “[59, 14, 17] accelerated fully connected layers with truncated SVD.”
• SVDNet for Pedestrian Retrieval (2017, methodology): “Truncated SVD [8, 27] is widely used for CNN model compression.”
• Conditional Teacher-student Learning (2019, unknown): “[30, 31], subspace-based [32, 33] and adversarial learning-based [34, 35] approaches.”