Attention for Fine-Grained Categorization

By Pierre Sermanet, Andrea Frome, Esteban Real. 2014

160 citations recorded by the source corpus.

This paper presents experiments extending the work of Ba et al. (2014) on recurrent neural models for attention into less constrained visual environments, specifically fine-grained categorization on the Stanford Dogs data set. In this work we use an RNN of the same structure but substitute a more powerful visual network and perform large-scale pre-training of the visual network outside of the attention RNN. Most work in attention models to date focuses on tasks with toy or more constrained visual environments, whereas we present results for fine-grained categorization better than the state-of-the-art GoogLeNet classification model. We show that our model learns to direct high resolution attention to the most discriminative regions without any spatial supervision such as bounding boxes, and it is able to discriminate fine-grained dog breeds moderately well even when given only an initial low-resolution context image and narrow, inexpensive glimpses at faces and fur patterns. This and similar attention models have the major advantage of being trained end-to-end, as opposed to other current detection and recognition pipelines with hand-engineered components where information is lost. While our model is state-of-the-art, further work is needed to fully leverage the sequential input.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Spatial Transformer Networks (2015, background): “In this section we discuss the prior work related to the paper, covering the central ideas of modelling transformations with neural networks [12, 13, 27], learning and analysing transformation-invariant representations [3, 5, 8, 17, 19, 25], as well as attention and detection mechanisms for feature ”
DRAW: A Recurrent Neural Network For Image Generation (2015, background): “…visual structure can be better captured by a sequence of partial glimpses, or foveations, than by a single sweep through the entire image (Larochelle & Hinton, 2010; Denil et al., 2012; Tang et al., 2013; Ranzato, 2014; Zheng et al., 2014; Mnih et al., 2014; Ba et al., 2014; Sermanet et al., 2014).”
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, methodology): “work is [46], which uses an attention-based recurrent neural network to achieve 76.”
Concept whitening for interpretable image recognition (2020, background): “, 2019; Wu & Song, 2019), to do classification based on hard attention (Mnih et al., 2014; Ba et al., 2014; Sermanet et al., 2015; Elsayed et al., 2019), or to do image recognition by decomposing the components of images (Saralajew et al.”
Learning to Skim Text (2017, methodology): “Closely related to our work is the idea of learning visual attention with neural networks (Mnih et al., 2014; Ba et al., 2014; Sermanet et al., 2014), where a recurrent model is used to combine visual evidence at multiple ﬁxations processed by a convolutional neural network.”
Neural Network Interpretation via Fine Grained Textual Summarization (2018, background): “Attention based models (Sermanet, Frome, and Real 2015) works by focusing attention to discriminative parts of an image object.”
Fully Convolutional Attention Localization Networks: Efficient Attention Localization for Fine-Grained Recognition (2016, methodology): “[3] further extend [2] and present experiments on fine-grained recognition datasets.”
Recurrent Modeling of Interaction Context for Collective Activity Recognition (2017, background): “Recently, LSTM network is also applied in fine grained action detection [25], human trajectory prediction [1], and in object recognition in the context of recurrent visual attention [3] [23] [29].”
Recurrent Mixture Density Network for Spatiotemporal Visual Attention (2016, background): “Other work has been geared towards learning hard-attentional models, which explicitly ignore and discard parts of the input (Larochelle & Hinton, 2010; Bazzani et al., 2011; Denil et al., 2012; Mnih et al., 2014; Xu et al., 2015; Sermanet et al., 2014; Ba et al., 2015; Yoo et al., 2015; Zheng et al.”
A survey on deep learning-based fine-grained object classification and semantic segmentation (2017, methodology): “Inspired from the way how humans perform visual sequence recognition, such as reading by continually moving the fovea to the next relevant object or character, recognizing the individual object, and adding the recognized object to the internal representation of the sequence, the attention for fine-g”
Object-Part Attention Model for Fine-Grained Image Classification (2017, unknown): “[27] propose the attention for fine-grained”
Adapting Models to Signal Degradation using Distillation (2016, methodology): “those using novel model architectures[12, 25, 31, 41], and using additional annotations to train part and object detectors [4, 7, 52, 53], we perform experiments with simple models in the interest of a detailed analysis.”