Practical Variational Inference for Neural Networks

By Alex Graves. 2011

1,876 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, neutral): “075 [Graves, 2011].”
Generating Sequences With Recurrent Neural Networks (2013, methodology): “…weight noise [18] with a std. deviation of 0.075 applied to the network weights at the start of each training sequence, and adaptive weight noise [8], where the variance of the noise is learned along with the weights using a Minimum description Length (or equivalently, variational inference)…”
Speech recognition with deep recurrent neural networks (2013, background): “Weight noise tends to ‘simplify’ neural networks, in the sense of reducing the amount of information required to transmit the parameters [23, 24], which improves generalisation.”
Deep AutoRegressive Networks (2013, methodology): “We found that regularisation by adaptive weight noise on these small data sets (Graves, 2011) did not yield good results, but early stopping based on the performance on the validation set worked well.”
Neural Autoregressive Distribution Estimation (2016, methodology): “Adaptive weight noise (adaNoise) was either used or not to avoid the need for early stopping (Graves, 2011).”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “A number of recent works develop scaleable approximate Bayesian methods for neural networks, beginning with Graves (2011), Blundell et al. (2015), Kingma et al. (2015), Hernández-Lobato & Adams (2015), and Gal & Ghahramani (2016).”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “Residual dropout with rate 0.1 is applied, as is weight-noise [45] and L2-regularization with weight 1e-6.”
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, methodology): “First, we turn on a variational weight noise [27] of standard deviation 0.”
Multilingual Speech Recognition with a Single End-to-End Model (2017, methodology): “For regularization, we apply a small L2 weight penalty of 1e-6 and add Gaussian weight noise [23] with standard deviation of 0.”
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, methodology): “01) to model weights [25] starting at 10K steps.”
Light Gated Recurrent Units for Speech Recognition (2018, methodology): “The application of weight noise is a simplification of adaptive weight noise [64] and has been successfully used before to regularize CTC-LSTM models [65].”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “Moreover, adding noise to weights and activations when computing the parameter gradients provide a form of regularization that can help to generalize better, as previously shown with variational weight noise (Graves, 2011), Dropout (Srivastava et al., 2014) and DropConnect (Wan et al., 2013).”
Attention-Based Models for Speech Recognition (2015, methodology): “We used the adaptive weight noise as a main regularizer [22].”
On Using Monolingual Corpora in Neural Machine Translation (2015, background): “001 ) to each parameter to prevent overﬁtting (Graves, 2011).”
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (2015, methodology): “This posterior is not tractable in general, and we may use variational inference to approximate it (as was done in [23–25 , We next extend this approximate variational inference to probabilistic RNNs, and use a q ( ω ) distribution that will give rise to a new variant of dropout in RNNs.”
Recurrent Highway Networks (2016, positive): “Additive Gaussian weight noise [34] was added to prevent overﬁtting and iteratively increased to 0.03 in increments of 0.01.”
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, methodology): “An alternative approach to variational inference makes use of expectation propagation (Hern´andez-Lobato & Adams, 2015) and has improved considerably in RMSE and uncertainty estimation on VI approaches such as (Graves, 2011).”
How to Construct Deep Recurrent Neural Networks (2013, background): “075 to each weight parameter every time the gradient is computed (Graves, 2011).”