Learning Wake-Sleep Recurrent Attention Models

By Jimmy Ba, Ruslan Salakhutdinov, Roger Baker Grosse, Brendan J. Frey. 2015

65 citations recorded by the source corpus.

Despite their success, convolutional neural networks are computationally expensive because they must examine all image locations. Stochastic attention-based models have been shown to improve computational efficiency at test time, but they remain difficult to train because of intractable posterior inference and high variance in the stochastic gradient estimates. Borrowing techniques from the literature on training deep generative models, we present the Wake-Sleep Recurrent Attention Model, a method for training stochastic attention networks which improves posterior inference and which reduces the variability in the stochastic gradients. We show that our method can greatly speed up the training time for stochastic attention networks in the domains of image classification and caption generation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Neural Variational Inference for Text Processing (2015, background): “The instantiations of these ideas (Gregor et al., 2015; Kingma et al., 2014; Ba et al., 2015) have demonstrated strong performance on the tasks of image processing.”
Action Recognition using Visual Attention (2015, methodology): “…been shown to achieve promising results on several challenging tasks, including caption generation (Xu et al., 2015), machine translation (Bahdanau et al., 2015), game-playing and tracking (Mnih et al., 2014), as well as image recognition (e.g. Street View House Numbers dataset (Ba et al., 2015b)).”
Variational Inference for Monte Carlo Objectives (2016, methodology): “While this is infeasible for the models we are considering, this fact suggests that making the proposal distribution close to the posterior is a sensible strategy.”
PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, background): “Unlike recent works on visual attention [1, 11, 20] which consider rectangular crops of an image, PerforatedCNNs can process non-rectangular and even disjoint salient parts of the image by choosing appropriate perforation masks in the convolutional layers.”
Variational Memory Addressing in Generative Models (2017, methodology): “A similar approach has however been used for training stochastic attention for image captioning [2].”
Adaptive Feature Abstraction for Translating Video to Language (2016, methodology): “Inspired by importance sampling, the multi-sample stochastic lower bound has been recently used for latent variable models (Ba et al., 2015; Burda et al., 2016), deﬁned as where m 1 , . . . , m K are independent samples.”
One-Shot Generalization in Deep Generative Models (2016, methodology): “Attention has been most widely used for classification tasks, having been shown to improve both scalability and generalization (Larochelle & Hinton, 2010; Chikkerur et al., 2010; Xu et al., 2015; Jaderberg et al., 2015; Mnih et al., 2014; Ba et al., 2015).”
Saliency-based Sequential Image Attention with Multiset Prediction (2017, unknown): “This "hard attention" mechanism developed in [31] has since been widely used [27, 44, 35, 2].”
Adaptive Feature Abstraction for Translating Video to Text (2016, methodology): “ P m p(mjA)p(Yjm;A) P m p(mjA)logp(Yjm;A); (9) where m = fm tg t=1;:::;T. Inspired by importance sampling, the multi-sample stochastic lower bound has been recently used for latent variable models (Ba et al., 2015; Burda et al., 2016), deﬁned as LK(Y) = P m1:K p(m 1:KjA) h log 1 K P K k=1 p(Yjmk;A)”
Training Deep Generative Models: Variations on a Theme (2015, methodology): “Work in [5, 1, 6] can be interpreted as applications of our KL-based bound, using a “meta” auxiliary distribution formed by applying a normalized importance sampling correction to q.”
Textual Entailment with Structured Attentions and Composition (2016, background): “Furthermore, as MacCartney and Manning (2009) point out, the entailment relation between sentences is modular, and can be modeled as the composition of subtree-level entailment relations.”
Attention mechanism in neural networks: where it comes and where it goes (2022, background): “Wake-Sleep Recurrent Attention Model (WS-RAM) speeds up the training time for image classification and caption generation tasks [72].”
Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization (2017, background): “The importance weighted objective has been applied to various applications such as generative modeling [5, 7], training binary stochastic feed-forward networks [30] and training recurrent attention models [4].”
Deep Attention Recurrent Q-Network (2015, background): “That is why another auspicious direction of future research would be (i) to test different techniques for reducing stochastic gradient variability [13], (ii) to apply different approaches to training stochastic attention networks [14, 15].”
Call Attention to Rumors: Deep Attention Based Recurrent Neural Networks for Early Rumor Detection (2017, background): “…0.67 1.00 @Evertxn: Donald Trump has been disqualified from running for president xD 0.91 1.00 0.25 0.67 0.98 0.83 0.47 0.67 1.00 0.03 @MikeMinusJordan: Trump: Marco Rubio Disqualified, 'Cannot Be President' After 1.00 0.67 0.67 0.98 0.73 0.67 1.00 0.45 Refusing To Undo Obama Executive Amnesty…”
Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives (2018, methodology): “It has been applied to generative modeling of images (Ba et al., 2015).”
ZhuSuan: A Library for Bayesian Deep Learning (2017, methodology): “, 2017; Paige and Wood, 2016) and learning attention models (Ba et al., 2015).”
Latent Alignment and Variational Attention (2018, methodology): “Two notable exceptions are [4, 41]: both utilize amortized variational inference to learn a sampling distribution which is used obtain importance-sampled estimates of the log marginal likelihood [12].”