An analysis of noise in recurrent neural networks: convergence and generalization

By Kam-Chuen Jim, C. Lee Giles, Bill G. Horne. 1996

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Generating Sequences With Recurrent Neural Networks (2013, methodology): “Since both networks overfit the training data, we also experiment with two types of regularisation: weight noise [18] with a std. deviation of 0.075 applied to the network weights at the start of each training sequence, and adaptive weight noise [8], where the variance of the noise is learned along…”
Speech recognition with deep recurrent neural networks (2013, methodology): “Two regularisers are employed in this paper: early stopping and weight noise (the addition of zero-mean, fixed variance Gaussian noise to the network weights during training [22]).”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, background): “Variational noise [5, 30] is introduced to the model as a regularization.”
Sequence Transduction with Recurrent Neural Networks (2012, methodology): “3 visualises these relationships for an RNN transducer applied to ‘end-to-end’ speech recognition, where raw spectrogram images are directly transcribed with character sequences with no intermediate conversion into phonemes.”
Regularizing RNNs by Stabilizing Activations (2015, background): “Other regularizers for RNNs that do not target norm stability include weight noise (Jim et al., 1996) and dropout (Pham et al., 2013; Pachitariu & Sahani, 2013; Zaremba et al., 2014).”
Deep Learning and Its Applications in Biomedicine (2018, methodology): “Adding noise to the input or to the weights are efﬁcient regularization strategies [41,92], as in the case of a denoising AE [93].”
Regularizing Recurrent Networks - On Injected Noise and Norm-based Methods (2014, methodology): “In the following subsections, we use the additive and multiplicative noise addition model by Jim et al. (1996) to evaluate the performance of a recurrent network in terms of preserving long term dependencies in musical chord sequences.”
Practical Variational Inference for Neural Networks (2011, background): “In particular, if the prior P (α) is uniform and a single weight sample is taken for each element of D, then minimising L(α,β,D) is identical to minimising L (w,D) with weight noise or synaptic noise [13].”
On the energy landscape of deep networks (2015, methodology): “However, the mini-batch gradient often does not have high variance in all directions which necessitates additional gradient noise (Jim et al., 1996; Nalisnick et al., 2015) and makes SGD highly dependent on the dimensionality (Lee et al., 2016).”
Hybrid speech recognition with Deep Bidirectional LSTM (2013, unknown): “For some of the experiments Gaussian noise was added to the network weights during training [15].”
The Effect of Gradient Noise on the Energy Landscape of Deep Networks (2015, background): “The weight update equation in the presence of additive noise (Jim et al., 1996) can be written as σ ← σ −η ∇Hn,p(σ)−∆; where ∆ ∈ Rn is a zero-mean Gaussian random vector with variance ν2In×n, ∇Hn,p(σ) is the backpropagated gradient and η is the learning rate.”
Attention-Based End-to-End Speech Recognition on Voice Search (2017, methodology): “Thus two main regularizations are used in this paper: L2 regularization and Gaussian weight noise [18, 19].”