The application of two-level attention models in deep convolutional neural network for fine-grained image classification

By Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, Zheng Zhang. 2014

824 citations recorded by the source corpus.

Fine-grained classification is challenging because categories can only be discriminated by subtle and local differences. Variances in the pose, scale or rotation usually make the problem more difficult. Most fine-grained classification systems follow the pipeline of finding foreground object or object parts (where) to extract discriminative features (what). In this paper, we propose to apply visual attention to fine-grained classification task using deep neural network. Our pipeline integrates three types of attention: the bottom-up attention that propose candidate patches, the object-level top-down attention that selects relevant patches to a certain object, and the part-level top-down attention that localizes discriminative parts. We combine these attentions to train domain-specific deep nets, then use it to improve both the what and where aspects. Importantly, we avoid using expensive annotations like bounding box or part information from end-to-end. The weak supervision constraint makes our work easier to generalize. We have verified the effectiveness of the method on the subsets of ILSVRC2012 dataset and CUB200 2011 dataset. Our pipeline delivered significant improvements and achieved the best accuracy under the weakest supervision condition. The performance is competitive against other methods that rely on additional annotations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Fine-Grained Image Classification via Combining Vision and Language (2017, methodology): “Xiao et al. [2] propose a two-level attention model, which combines two level at-tentions to select relevant proposals to the object and the discriminative parts.”
Dynamic Neural Networks: A Survey (2021, neutral): “Sa & Sp Image segmentation [34], [129], [146], [148], [150], [154], [156], [216], [217], [218], [219], [220], Image-to-image translation [221], Object detection [110], [111], [147], [148], [164], Semantic image synthesis [222], [223], [224], Computer Image denoising [225], Fine-grained classificatio”
Attention to Scale: Scale-Aware Semantic Image Segmentation (2015, methodology): “Attention models for deep networks: In computer vision, attention models have been used widely used for image classification [8, 25, 53] and object detection [4, 7, 57].”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “In the same line, [278] presented a three-level attention model to perform a ﬁne-grained classiﬁ-cation task.”
Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, background): “…approaches have shown remarkable success in solving problems such as allowing natural language translation models to process words in an appropriate non-sequential order [55], and they have also been applied in domains such as ﬁne-grained image classiﬁcation [56] and visual question answering [57].”
Recent advances in convolutional neural networks (2015, methodology): “[111] applies visual attention in CNN for fine-grained classification.”
Residual Attention Network for Image Classification (2017, methodology): “the ground truth bounding boxes or detailed segmentation masks [6], unsupervised learning [35] is usually used to generate region proposals for image classification.”
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, methodology): “The use of existing test sets serves to provide an accurate measure of performance and put our work in a larger context, but results may not be strictly comparable with prior work that operates within a single given dataset.”
Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, background): “image classification and captioning [12, 13, 14, 15, 16, 17, 18, 19, 20], text classification [21, 22], generative models [6, 23, 24], object tracking [25], and reinforcement learning [26, 27, 28].”
Deep Image: Scaling up Image Recognition (2015, methodology): “% by adding bounding box annotations only at test time. Table 2. Species categorization results on CUB-200-2011 Method Training Testing Accura BBox Part BBox Part cy(%) Alignments [17] 53.6 Attention [49] 77.9 B-CNN (D,M) [30] 80.9 Part discovery [44] 81.0 Ours 84.9 GMTL [38] √ √ 44.2 Symbiotic [6] ”
ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs (2015, methodology): “, 2015), image classification (Xiao et al., 2015), caption generation (Xu et al.”
Neural Activation Constellations: Unsupervised Part Model Discovery with Convolutional Networks (2015, background): “[36] by 3.1%, an error decrease of more than 16%.”
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet (2019, neutral): “Xiao et al. (2015) proposed a multi-step approach combining object, part-and domain-detectors to reach a classiﬁcation decision.”
The Devil is in the Tails: Fine-grained Classification in the Wild (2017, background): “Most fine-grained research papers present a novel model for classification [4, 6, 7, 12, 15, 17, 25, 31, 40, 45, 46, 57, 58, 59, 61, 62].”
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks (2021, methodology): “Common approaches to explainability of deep learning methods in computer vision are saliency maps [1], attention maps [47], and activation vectors [25].”
Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning (2018, methodology): “To capture subtle visual differences, visual attention [60, 16, 69] and deep metric learning [45, 10] are often used.”
Do Semantic Parts Emerge in Convolutional Neural Networks? (2016, background): “Several works use ﬁlter responses for recognition tasks [16,26,27,28,34].”