Recurrent Models of Visual Attention
By Volodymyr Mnih, Nicolas Manfred Otto Heess, Alex Graves, Koray Kavukcuoglu. 2014
3,987 citations recorded by the source corpus.
Applying convolutional neural networks to large images is computationally expensive because the amount of computation scales linearly with the number of image pixels. We present a novel recurrent neural network model that is capable of extracting information from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution. Like convolutional neural networks, the proposed model has a degree of translation invariance built-in, but the amount of computation it performs can be controlled independently of the input image size. While the model is non-differentiable, it can be trained using reinforcement learning methods to learn task-specific policies. We evaluate our model on several image classification tasks, where it significantly outperforms a convolutional neural network baseline on cluttered images, and on a dynamic visual control problem, where it learns to track a simple object without an explicit training signal for doing so.
Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Model Architectures, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cognitive Neuroscience
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Nicolas Manfred Otto Heess
- J. Urgen Schmidhuber
- Chelsea Finn
- Josh Tenenbaum
- Jianbing Shen
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Kyunghyun Cho
- Ruslan Salakhutdinov
- Richard Socher
- Mohit Bansal
- J. Urgen Schmidhuber
- Jianfeng Gao
In Human-Computer Interaction
- Xiaogang Wang
- Wanli Ouyang
- Jitendra Malik
- G. Wang
- J. Urgen Schmidhuber
- Alan L. Yuille
- Xiaoou Tang
- Deva Ramanan
Citations in the corpus
- Effective Approaches to Attention-based Neural Machine Translation (2015, background): “…allowing models to learn alignments between different modalities, e.g., between image objects and agent actions in the dynamic control problem (Mnih et al., 2014), between speech frames and text in the speech recognition task (Chorowski et al., 2014), or between visual features of a picture…”
- Teaching Machines to Read and Comprehend (2015, methodology): “These models draw on recent developments for incorporating attention mechanisms into recurrent neural network architectures [6, 7, 8].”
- Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “Systems that produce explanations either in the form of the language (Ling et al., 2017; Hendricks et al., 2016), attention (Bahdanau et al., 2014; Mnih et al., 2014; Gulcehre et al., 2016; Malinowski et al., 2018; Xu and Saenko, 2016; Yang et al., 2016), phrase selection (Lei et al., 2016),…”
- Object-Centric Learning with Slot Attention (2020, background): “[69] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al.”
- TensorFlow: A system for large-scale machine learning (2016, neutral): “Using a separate, less familiar programming language for implementing layers is a barrier for machine learning researchers who seek to experiment with new layer architectures, such as sampled softmax classifiers [37] and attention modules [53].”
- Learning to decompose for object detection and instance segmentation (2015, methodology): “We acknowledge support from NSF 1446631, 1452851 and from NVIDIA for GPU hardware.”
- Gradient Estimation Using Stochastic Computation Graphs (2015, methodology): “Combining ideas from from those two perennial topics, recent models of attention [15] and memory [29] have used networks that involve a combination of stochastic and deterministic operations.”
- DRAW: A Recurrent Neural Network For Image Generation (2015, methodology): “The network is similar to the recently introduced Recurrent Attention Model (RAM) (Mnih et al., 2014), except that our attention method is differentiable; we therefore refer to it as “Differ-entiable RAM”.”
- Dynamic Neural Networks: A Survey (2021, methodology): “The most typical approach is formulating a classification task as a sequential decision process, and adopting RNNs to make iterative predictions based on selected patches [159], [160].”
- OpenSpiel: A Framework for Reinforcement Learning in Games (2019, background): “Game Reference(s) Status Backgammon Wikipedia Breakthrough Wikipedia Bridge bidding Wikipedia Coin Game [56] ∼ Connect Four Wikipedia Cooperative Box-Pushing [62] ∼ Chess Wikipedia First-price Sealed-bid Auction Wikipedia Go Wikipedia Goofspiel Wikipedia Havannah Wikipedia Hex Wikipedia ∼ Kuhn poker”
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “, 2014) and object recognition (Ba et al., 2014; Mnih et al., 2014), we investigate models that can attend to salient part of an image while generating its caption.”
- Squeeze-and-Excitation Networks (2017, background): “Attention can be viewed, broadly, as a tool to bias the allocation of available processing resources towards the most informative components of an input signal [17, 18, 22, 29, 32].”
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, background): “Still, reasoning about discrete computation comes naturally to humans, and so, despite the difficulty associated, many modern architectures incorporate discrete stochasticity (Mnih et al., 2014; Xu et al., 2015; Kočiský et al., 2016).”
- Rationalizing Neural Predictions (2016, methodology): “space of possible rationales in terms of their interaction with the encoder. We could also apply variance reduction techniques to increase stability of stochastic training (cf. (Weaver and Tao, 2001; Mnih et al., 2014; Ba et al., 2015; Xu et al., 2015)). 7 Acknowledgments We thank Prof. Julian McAul”
- ReasoNet: Learning to Stop Reading in Machine Comprehension (2016, methodology): “Unlike previous approaches using a xed number of hops or iterations, motivated by [15, 16], we propose a termination module in the inference.”
- Gated-Attention Readers for Text Comprehension (2016, background): “The attention mechanism has been introduced recently to model human focus, leading to significant improvement in machine translation and image captioning (Bahdanau et al., 2014; Mnih et al., 2014).”