Bayesian Recurrent Neural Networks

By Meire Fortunato, Charles Blundell, Oriol Vinyals. 2017

209 citations recorded by the source corpus.

In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80\%. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “…by Bayes by Backprop trained neural networks has been used successfully for training feedforward neural networks in both supervised and reinforcement learning environments [4, 42, 28], for training recurrent neural networks [11], but has not been applied to convolutional neural networks to-date.”
Memory-based Parameter Adaptation (2018, background): “This can also be seen as posterior sharpening (Fortunato et al., 2017), where we can think of the second term as an approximation of log p ( y t | x t , θ ) .”
Noisy Networks for Exploration (2017, background): “A similar randomisation technique can also be applied to LSTM units [10] and is easily extended to reinforcement learning, we leave this as future work.”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, methodology): “This is representative of commonly used dynamic evaluation methods (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017).”
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning (2018, methodology): “Dynamic evaluation algorithms (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017), for example, learn an approximate global distribution at training time and adapt those model parameters at test time to fit the current local distribution via gradient descent.”
Analyzing the role of model uncertainty for electronic health records (2019, neutral): “Clinical Metrics.”
Learning to Adapt: Meta-Learning for Model-Based Control (2018, unknown): “Dynamic evaluation algorithms (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017), for example, learn an approximate global distribution at train time and adapt those model parameters at test time to fit the current local distribution via gradient descent.”
Actively Learning what makes a Discrete Sequence Valid (2017, methodology): “We propose learning a Bayesian Recurrent Neural Network (Gal, 2015; Fortunato et al., 2017) to approximate these probabilities, given access to a function that labels full sequences as valid/invalid (such functions already exist for molecules, symbolic expressions, and many natural language…”
U-CAM: Visual Explanation Using Uncertainty Based Class Activation Maps (2019, unknown): “The work by [15, 12] estimates model uncertainty of the deep network (CNN, RNN) with the help of dropout [45].”
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (2019, background): “Tempering is common in modern Bayesian deep learning, for both variational inference and MCMC approaches (e.g., Li et al., 2016a; Nguyen et al., 2017; Fortunato et al., 2017).”
Deep and Confident Prediction for Time Series at Uber (2017, background): “Most approaches are based on variational inference that optimizes the variational lower bound, including stochastic search [14], variational Bayes [15], probabilistic backpropagation [16], Bayes by BackProp [17] and its extension [18].”
Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “Recently, the uncertainty afforded by Bayes by Backprop trained neural networks has been used successfully to train feedforward neural networks in both supervised and reinforcement learning environments [5, 7, 8], for training recurrent neural networks [9], and for CNNs [10, 11].”
A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU (2023, methodology): “To overcome this, more advanced RNN variants have been developed, including Long Short-Term Memory (LSTM) [ 32 ], bidirectional LSTM [ 33 ], Gated Recurrent Unit (GRU) [ 34 ], bidirectional GRU [ 35 ], Bayesian RNN [ 36 ], and others.”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, background): “Figure 1 implements a Bayesian RNN; Appendix A implements a Bayesian CNN (ResNet-50).”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “The authors in [119] investigated a variational Bayes scheme for RNN.”
Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “As an add-on method to further enhance the stability of the optimization, posterior sharpening [5] could be applied to Bayesian C NN s in future work.”
Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning (2019, background): “, Fortunato et al. [2018], achieve a similar result just by ignoring the KL to the prior in the loss so that the weight variances tend to shrink to overfit the training data.”