Learning Generative Models with Visual Attention

By Yichuan Tang, Nitish Srivastava, Ruslan Salakhutdinov. 2013

87 citations recorded by the source corpus.

Attention has long been proposed by psychologists to be important for efficiently dealing with the massive amounts of sensory stimulus in the neocortex. Inspired by the attention models in visual neuroscience and the need for object-centered data for generative models, we propose a deep-learning based generative framework using attention. The attentional mechanism propagates signals from the region of interest in a scene to an aligned canonical representation for generative modeling. By ignoring scene background clutter, the generative model can concentrate its resources on the object of interest. A convolutional neural net is employed to provide good initializations during posterior inference which uses Hamiltonian Monte Carlo. Upon learning images of faces, our model can robustly attend to the face region of novel test subjects. More importantly, our model can learn generative models of new faces from a novel dataset of large images where the face locations are not known.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

DRAW: A Recurrent Neural Network For Image Generation (2015, background): “…visual structure can be better captured by a sequence of partial glimpses, or foveations, than by a single sweep through the entire image (Larochelle & Hinton, 2010; Denil et al., 2012; Tang et al., 2013; Ranzato, 2014; Zheng et al., 2014; Mnih et al., 2014; Ba et al., 2014; Sermanet et al., 2014).”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “com/kelvinxu/ arctic-captions 3 We verified that our BLEU evaluation code matches the authors of Vinyals et al. (2014), Karpathy & Li (2014) and Kiros et al.”
Object Detectors Emerge in Deep Scene CNNs (2014, background): “…have shown that one can detect objects by applying the network multiple times in different locations (Girshick et al., 2014), or focusing attention (Tang et al., 2014), or by doing segmentation (Grangier et al., 2009; Farabet et al., 2013), here we show that the same network can do both object…”
One-Shot Generalization in Deep Generative Models (2016, methodology): “Gregor et al. (2015) use a generative attention based on Gaussian convolutions parameterized by a location and scale, and Tang et al. (2014) use 2D similarity transformations.”
Image Captioning with Semantic Attention (2016, methodology): “In terms of models, [21, 33] approach it with Boltzmann machines while [28] does with recurrent neural networks.”
On the Binding Problem in Artificial Neural Networks (2020, background): “Similarly, Tang et al. (2014) incorporates a window of attention in a deep belief network to extract object representations by performing (stochastic) inference over the window parameters alongside the belief states.”
Attend, Infer, Repeat: Fast Scene Understanding with Generative Models (2016, background): “…range from highly speciﬁed (Milch et al., 2005; Mansinghka et al., 2013), to partially speciﬁed (Zhu & Mumford, 2006; Roux et al., 2011; Heess et al., 2011; Eslami & Williams, 2014; Tang et al., 2013; 2014), to largely unspeciﬁed (Hinton, 2002; Salakhutdinov & Hin-ton, 2009; Eslami et al., 2012).”
Not All Contexts Are Created Equal: Better Word Representations with Variable Attention (2015, background): “Attention models are common in vision related tasks (Tang et al., 2014), where models learn to pay attention to certain parts of a image in order to make accurate predictions.”
Attention for Fine-Grained Categorization (2014, background): “…(Mnih et al., 2014), expression classiﬁcation for 100 × 100 aligned faces (Larochelle & Hinton, 2010; Zheng et al., 2014), detection of frontal faces (Tang et al., 2013), tracking of hockey players (Bazzani et al., 2011; Denil et al., 2012) and gesture recognition (Darrell & Pentland, 1996).”
Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network (2015, background): “Their objective is to learn the attention mechanism that can adaptively focus on salient part of an image or video for various computer vision tasks, such as object recognition [1, 2, 18], object tracking [3], caption generation [37], image generation [35], etc.”
Attention mechanism in neural networks: where it comes and where it goes (2022, background): “Additionally, a deep learning-based framework using attention has been proposed for generative modeling [43].”
Learning Wake-Sleep Recurrent Attention Models (2015, methodology): “This has motivated recent work on visual attention-based models [2, 3, 4], which reduce the number of parameters and computational operations by selecting informative regions of an image to focus on.”
Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge (2019, background): “They also boost deep model performance by guiding the generators to focus on the relevant information [43, 42, 40, 25, 37, 13, 3, 26, 12, 14, 32].”
The Geometric Foundations of Hamiltonian Monte Carlo (2014, unknown): “Hamiltonian Monte Carlo [20,52] has proven tremendously successful at extracting inferences from these models, with applications spanning computer science [69,70], ecology [63,71], epidemiology [16], linguistics [37], pharmacokinetics [76], physics [39,58,62, 74], and political science [28], to name”
Deep neural networks: a new framework for modelling biological vision and brain information processing (2015, background): “…automatic rapid interpretation of even static visual images (Sugase et al. 1999, Brincat & Connor 2006, Freiwald & Tsao 2010, Carlson et al. 2013, Cichy et al. 2014; Tang et al. 2014a), leading to the emergence of representations that clearly distinguish particular objects and object categories.”
Learning High-level Prior with Convolutional Neural Networks for Semantic Segmentation (2015, background): “However, with the rapid research advancement in advanced network structure such as [33, 13, 1] that introduces visual attention or scale invariance, a more compact cod-”
On the convergence of Hamiltonian Monte Carlo (2017, result): “Since then, it has been consistently reported in numerous works that HMC achieves better performance than more classical MCMC methods; see among many others [17, 7, 2, 16, 12] and the references therein.”
Training a Feedback Loop for Hand Pose Estimation (2015, background): “Our approach is related to generative approaches [3], in particular [32] which also features a feedback loop reminiscent of ours.”
Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet (2014, background): “Extending state-of-the-art neural networks with attention is an exciting new direction of research (Tang et al., 2014; Mnih et al., 2014).”