Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling
By Zhe Gan, Chunyuan Li, Changyou Chen, Yunchen Pu, Qinliang Su, Lawrence Carin. 2016
42 citations recorded by the source corpus.
Recurrent neural networks (RNNs) have shown promising performance for language modeling. However, traditional training of RNNs using back-propagation through time often suffers from overfitting. One reason for this is that stochastic optimization (used for large training sets) does not provide good estimates of model uncertainty. This paper leverages recent advances in stochastic gradient Markov Chain Monte Carlo (also appropriate for large training sets) to learn weight uncertainty in RNNs. It yields a principled Bayesian learning algorithm, adding gradient noise during training (enhancing exploration of the model-parameter space) and model averaging when testing. Extensive experiments on various RNN models and across a broad range of applications demonstrate the superiority of the proposed approach relative to stochastic optimization.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Chunyuan Li
- Sergey Levine
- Andrew Gordon Wilson
- Changyou Chen
- Jianfeng Gao
- Ruslan Salakhutdinov
- Yoshua Bengio
- Quoc V. Le
Citations in the corpus
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (2019, background): “( ii ) In practice, empirical successes have been reported by training DNNs in relatively short time (Li et al., 2016b; Chen et al., 2014; Gan et al., 2016; Neelakantan et al., 2016; Saatchi & Wilson, 2017).”
- Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “In the traditional factor analysis (Gan et al., 2017c), the linear relation between x and z is assumed, and inference is done using mean field VB or Gibbs sampling.”
- Bayesian Recurrent Neural Networks (2017, methodology): “Recent approaches either attempt to justify dropout (Srivastava et al., 2014) and weight decay as a variational inference scheme (Gal & Ghahramani, 2016), or apply Stochastic Gradient Langevin dynamics (Welling & Teh, 2011, SGLD) to truncated back-propagation in time directly (Gan et al., 2016).”