Exploiting sparseness in deep neural networks for large vocabulary speech recognition

By Dong Yu, Frank Torsten Bernd Seide, Gang Li, Li Deng. 2012

156 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Exploring Sparsity in Recurrent Neural Networks (2017, neutral): “Another approach for pruning feed forward neural networks for speech recognition is using simple threshold to prune all weights (Yu et al., 2012) at a particular epoch.”
Fixed-point feedforward deep neural network design using weights +1, 0, and −1 (2014, background): “Recently, a research work that tries to increase the sparseness of the weights by pruning out small valued ones has been developed in order to reduce the model size and the execution time with a CPU [9].”
Deep Rewiring: Training very sparse deep networks (2017, methodology): “There are however several heuristic methods for pruning a larger network (Han et al., 2015b;a; Collins & Kohli, 2014; Yang et al., 2015; Srinivas & Babu, 2015), that is, the network is ﬁrst trained to convergence, and network connections and / or neurons are pruned only subsequently.”
Feature Learning in Deep Neural Networks - A Study on Speech Recognition Tasks (2013, methodology): “In all the following discussions, we use DNNs in the framework of the CD-DNN-HMM [6–10] and use speech recognition as our classiﬁcation task.”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, methodology): “The most current implementation of this approach is based on multiple layers of neural networks using backpropagation learning [135].”
Resiliency of Deep Neural Networks under Quantization (2015, background): “Pruned ﬂoating-point weights are also utilized for efﬁcient GPU based implementations, where small valued weights are forced to zero to reduce the number of arithmetic operations and the memory space for weight storage (Yu et al., 2012b; Han et al., 2015).”
Large Vocabulary Speech Recognition Using Deep Tensor Neural Networks (2012, methodology): “CD-DNN-HMMs have achieved 16% [1][2] and 33% [3][4][5][6] relative recognition error reduction over strong, discriminatively trained CD-GMM-HMMs, respectively, on a voice search (VS) task [10] and the Switchboard (SWB) phone-call transcription task [11].”
Investigation on dimensionality reduction of concatenated features with deep neural network for LVCSR systems (2012, background): “…In these few years, the promising hybrid acoustic model, so-called context-dependent deep neural network hidden Markov models (CD-DNN-HMMs), has been greatly developed and achieved significant improvements on various large vocabulary continuous speech recognition (LVCSR) tasks [1][2][3][6][7].”
"Learning-Compression" Algorithms for Neural Net Pruning (2018, unknown): “[33, 15]), but this is timeconsuming in practice.”
Block-Sparse Recurrent Neural Networks (2017, methodology): “Yu et al. (2012) use a hard threshold to prune deep learning models.”
Less Is More: Towards Compact CNNs (2016, background): “Computing the gradient after certain number of iterations would make the learned parameters more stable [29].”
The Deep Tensor Neural Network With Applications to Large Vocabulary Speech Recognition (2013, methodology): “The features, lexicon and LM used in this study are the same as those used in our earlier work [4]–[6].”
Locally-connected and convolutional neural networks for small footprint speaker recognition (2015, background): “Previous works have taken note of DNN sparsity and attempted to train networks that are less sparse [12], or iteratively prune low-value weights [13].”
Nexus: a GPU cluster engine for accelerating DNN-based video analysis (2019, background): “Much work has focused on producing faster models often at small losses in accuracy [2, 31, 41] Further, models of varying accuracy can be combined to maintain high accuracy and performance [6, 14, 17, 20, 35].”
FACTORIZED DEEP NEURAL NETWORKS FOR ADAPTIVE SPEECH RECOGNITION (2012, background): “They have achieved 16% [4][5] and 33% [6][7][8] relative recognition error reduction over strong, discriminatively trained CD-GMM-HMMs, respectively, on a voice search (VS) task [9] and the Switchboard (SWB) phone-call transcription task [10].”
Machine Learning Paradigms for Speech Recognition: An Overview (2013, background): “The experimental results in [271] on a large scale ASR task show not only the DNN model size is reduced by 66% to 88%, the error rate is also slightly reduced by 0.”
MCDNN: An Approximation-Based Execution Framework for Deep Stream Processing Under Resource Constraints (2016, methodology): “An intermediate approach that is both automatic and effective is what we term model optimization, which are techniques that apply automatically to any DNN and reduce memory [13, 22, 50, 51] and processing demands [25, 42] of DNNs, typically at the cost of some classification accuracy.”
Improving low-resource CD-DNN-HMM using dropout and multilingual DNN training (2013, methodology): “Although speeding up model training greatly, these methods fail to improve recognition performance significantly [4].”