Learning Where to Attend with Deep Architectures for Image Tracking

By Misha Denil, Loris Bazzani, Hugo Larochelle, Nando de Freitas. 2011

210 citations recorded by the source corpus.

We discuss an attentional model for simultaneous object tracking and recognition that is driven by gaze data. Motivated by theories of perception, the model consists of two interacting pathways, identity and control, intended to mirror the what and where pathways in neuroscience models. The identity pathway models object appearance and performs classification using deep (factored)-restricted Boltzmann machines. At each point in time, the observations consist of foveated images, with decaying resolution toward the periphery of the gaze. The control pathway models the location, orientation, scale, and speed of the attended object. The posterior distribution of these states is estimated with particle filtering. Deeper in the control pathway, we encounter an attentional mechanism that learns to select gazes so as to minimize tracking uncertainty. Unlike in our previous work, we introduce gaze selection strategies that operate in the presence of partial information and on a continuous action space. We show that a straightforward extension of the existing approach to the partial information setting results in poor performance, and we propose an alternative method based on modeling the reward surface as a gaussian process. This approach gives good performance in the presence of partial information and allows us to expand the action space from a small, discrete set of fixation points to a continuous domain.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

DRAW: A Recurrent Neural Network For Image Generation (2015, background): “…structure can be better captured by a sequence of partial glimpses, or foveations, than by a single sweep through the entire image (Larochelle & Hinton, 2010; Denil et al., 2012; Tang et al., 2013; Ranzato, 2014; Zheng et al., 2014; Mnih et al., 2014; Ba et al., 2014; Ser-manet et al., 2014).”
Recurrent Models of Visual Attention (2014, background): “Our work is perhaps the most similar to the other attempts to implement attentional processing in a deep learning framework [6, 14, 17].”
Image Captioning with Semantic Attention (2016, background): “[6] studies it for image tracking, [1] studies it for image recognition of multiple objects, and [15] uses for image generation.”
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer (2016, methodology): “Early work on attention based tracking Larochelle & Hinton (2010), Denil et al. (2012) was motivated by human attention mechanism theories Rensink (2000) and was done via Restricted Bolz-mann Machines.”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “6) Automatic machine learning and hyperparameter tuning: In this application, the goal is to automatically select the best model (e.g., random forests, support vector machines, neural networks, etc.) and its associated hyperparameters for solving a task on a given dataset.”
Deep Fried Convnets (2014, background): “…efficient and preserves the flexibility of fully connected layers for feature transfer (cf. Lin et al., 2014; Szegedy et al., 2014), it realizes memory savings both during training and a test time (cf. Collins & Kohli, 2014), and does not require layerwise pretraining (cf. Denil et al., 2012).”
Deep Networks with Internal Selective Attention through Feedback Connections (2014, background): “In the context of computer vision, RL has been shown to be able to learn saccades in visual scenes to learn selective attention [40, 41], learn feedback to lower levels [42, 43], and improve face recognition [44, 45].”
Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks (2015, background): “Earlier works on utilizing the attention mechanism, both content-based and location-based, for object recognition/tracking can be found in [69], [70], [71].”
Learning Generative Models with Visual Attention (2013, background): “Several other related models attempt to learn where to look for objects [19, 20] and for video based tracking [21].”
Fully-Convolutional Siamese Networks for Object Tracking (2016, background): “[27] track objects with a particle filter that uses a learnt distance metric to compare the current appearance to that of the first frame.”
Attention for Fine-Grained Categorization (2014, methodology): “…such as detecting simple shapes (Schmidhuber & Huber, 1991), tasks based on MNIST digits (Larochelle & Hinton, 2010; Bazzani et al., 2011; Denil et al., 2012; Ranzato, 2014; Mnih et al., 2014), the vision-control game of “catch” (Mnih et al., 2014), expression classiﬁcation for 100 ×…”
Learning to Relate Images (2011, neutral): “22.”
Building a Regular Decision Boundary with Deep Networks (2017, background): “For example, max pooling [14], spatial transformers [13], local contrast normalization [15], attention networks [8], resnet [11], make the mathematical analysis even more difficult, since the way they are combined is mainly defined by a complex engineering process of trial and error based on the fin”
WiFi CSI Based Passive Human Activity Recognition Using Attention Based BLSTM (2019, methodology): “The attention model was ﬁrst designed for image recognition [24].”
Recurrent Attention Models for Depth-Based Person Identification (2016, background): “By measuring the sensitivity of output variables to variances in the input, attention models applied to image classification [76, 25, 70], image captioning [21, 71, 14], object detection [11], and tracking [18] have demystified many aspects of convolutional and recurrent networks.”