Reshaping deep neural network for fast decoding by node-pruning

By Tianxing He, Yuchen Fan, Yanmin Qian, Tian Tan, Kai Yu. 2014

121 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Dong Yu
3. Jinyu Li
4. Li Deng
5. Yoshua Bengio
6. Alexander M. Rush
7. Mohit Bansal
8. Yonatan Belinkov

In Efficiency & Compression

1. Jinyu Li
2. Xinchao Wang
3. Yifan Gong
4. Yanmin Qian
5. Kai Yu
6. Alexander M. Rush
7. Yoon Kim
8. Simon Lacoste-Julien

In Signal Processing

1. Jinyu Li
2. Yifan Gong
3. Dong Yu
4. Yanmin Qian
5. Kai Yu
6. Alexander M. Rush
7. Yoon Kim
8. Xinchao Wang

Citations in the corpus

• Sequence-Level Knowledge Distillation (2016, background): “Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas, November 1-5, 2016. c©2016 Association for Computational Linguistics”
• Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Some works prune neurons/channels individually based on some importance score [He et al., 2014, Li et al., 2017, Liebenwein et al., 2020, Mussay et al., 2020, 2021, Molchanov et al., 2017, Srinivas and Babu, 2015].”
• Automatic node selection for Deep Neural Networks using Group Lasso regularization (2016, methodology): “It resembles the weight-grouping-based norm that we use in our proposed training procedure [7].”
• Compression of Deep Learning Models for Text: A Survey (2020, background): “[31] proposed three node importance functions to determine importance score for neurons.”
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle (2018, background): “While currently the network architectures leading to state-of-the-art performance in various classification tasks are highly over-parameterized, it has also been observed that a major portion of the network parameters can be pruned without significant deterioration in performance [14]– [16] and it h”
• Scalpel: Customizing DNN pruning to the underlying hardware parallelism (2017, unknown): “[23] introduce a DNN reshape method which measures the importance of each node with a certain importance function.”
• DeepX: A Software Accelerator for Low-Power Deep Learning Inference on Mobile Devices (2016, methodology): “∙ Runtime Optimization: Various methods for optimizing deep learning models prior to execution [18], [19], [21] while useful, are insufficient.”
• Small-footprint high-performance deep neural network-based speech recognition using split-VQ (2015, background): “Index Terms— DNN, on device speech recognition, model compression, split-VQ”
• Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions (2018, background): “The recent work (Abbasi-Asl & Yu, 2017) iteratively pruned filters based on the classification accuracy reduction index, and achieved substantially higher classification accuracy compared to other structural compression schemes, e.g., (He et al., 2014; Li et al., 2016).”
• Knowledge Distillation for Sequence Model (2018, background): “Thus researches have been conducted on transferring the learned models to lowresource platforms [3, 4, 5, 6].”
• Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype (2018, background): “Node pruning (He et al., 2014), weight matrix decomposition (Xue et al., 2013), and filter separation (Bhattacharya and Lane, 2016) are complementary ways to create sparser DNNs with reduced computing requirements.”
• On Compressing Deep Models by Low Rank and Sparse Decomposition (2017, methodology): “[10] used measures like l1 norm of out-linked weights to identify unimportant neurons.”