Sparse connection and pruning in large dynamic artificial neural networks

By Nikko Ström. 1997

This paper presents new methods for training large neural networks for phoneme probability estimation. A combination of the time-delay architecture and the recurrent network architecture is used to capture the important dynamic information of the speech signal. Motivated by the fact that the number of connections in fully connected recurrent networks grows super-linear with the number of hidden units, schemes for sparse connection and connection pruning are explored. It is found that sparsely connected networks outperform their fully connected counterparts with an equal or smaller number of connections. The networks are evaluated in a hybrid HMM/ANN system for phoneme recognition on the TIMIT database. The achieved phoneme error-rate, 28.3%, for the standard 39 phoneme set on the core testset of the TIMIT database is not far from the lowest reported. All training and simulation software used is made freely available by the author, making reproduction of the results feasible.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “We use iterative magnitude pruning (Ström, 1997; Han et al., 2016): we iteratively zero out the smallest magnitude parameters and continue finetuning the model on the downstream task to recover lost accuracy.”
The State of Sparsity in Deep Neural Networks (2019, methodology): “Simple heuristics based on removing small magnitude weights have demonstrated high compression rates with minimal accuracy loss (Ström, 1997; Collins & Kohli, 2014; Han et al., 2015), and further refinement of the sparsification process for magnitude pruning techniques has increased achievable compr”
Rigging the Lottery: Making All Tickets Winners (2019, methodology): “Like many prior works (Thimm & Fiesler, 1995; Str¨om, 1997; Narang et al., 2017; Han et al., 2015), we drop parameters with the smallest magnitude.”
Characterising Bias in Compressed Models (2020, background): “Despite the widespread use of compression techniques, articulating the trade-offs of compression has overwhelming centered on change to overall accuracy for a given level of compression (Ström, 1997; Cun et al., 1990; Evci et al., 2019; Narang et al., 2017).”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, background): “…from a larger teacher model (model distillation) (Hinton et al., 2015) and finally pruning by setting a subset of weights or filters to zero (Louizos et al., 2017; Wen et al., 2016; Cun et al., 1990; Hassibi et al., 1993a; Ström, 1997; Hassibi et al., 1993b; See et al., 2016; Narang et al., 2017).”
What Do Compressed Deep Neural Networks Forget (2019, methodology): “…model (model distillation) (Hinton et al., 2015) and ﬁnally pruning by setting a subset of weights or ﬁlters to zero (Louizos et al., 2017; Wen et al., 2016; Cun et al., 1990; Hassibi et al., 1993b; Ström, 1997; Hassibi et al., 1993a; Zhu & Gupta, 2017; See et al., 2016; Narang et al., 2017).”
Channel-Level Acceleration of Deep Face Representations (2015, methodology): “Earlier implementations of network pruning [20]–[23] are used to improve model generalization and size while not targeting CNN run-time.”
Scalable distributed DNN training using commodity GPU cloud computing (2015, background): “It is well-known that typically the weights of a fully connected DNN are sparsely distributed with most weights close to zero [17] [18], so it is not surprising that subgradients are also sparse.”