Part-Stacked CNN for Fine-Grained Visual Categorization

By Shaoli Huang, Zhe Xu, Dacheng Tao, Ya Zhang. 2015

464 citations recorded by the source corpus.

In the context of fine-grained visual categorization, the ability to interpret models as human-understandable visual manuals is sometimes as important as achieving high classification accuracy. In this paper, we propose a novel Part-Stacked CNN architecture that explicitly explains the finegrained recognition process by modeling subtle differences from object parts. Based on manually-labeled strong part annotations, the proposed architecture consists of a fully convolutional network to locate multiple object parts and a two-stream classification network that encodes object-level and part-level cues simultaneously. By adopting a set of sharing strategies between the computation of multiple object parts, the proposed architecture is very efficient running at 20 frames/sec during inference. Experimental results on the CUB-200-2011 dataset reveal the effectiveness of the proposed architecture, from multiple perspectives of classification accuracy, model interpretability, and efficiency. Being able to provide interpretable recognition results in realtime, the proposed method is believed to be effective in practical applications.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Fine-Grained Image Classification via Combining Vision and Language (2017, methodology): “Models of ﬁne-grained image classiﬁcation have made great progress in recent years[1, 2, 3, 4, 5], due to the progress of deep neural networks.”
Concept Bottleneck Models (2020, background): “…concepts commonly use them in auxiliary objectives in a multi-task setup, or as auxiliary features; examples include using object parts (Huang et al., 2016; Zhou et al., 2018), parse trees (Zelenko et al., 2003; Bunescu & Mooney, 2005), or natural language explanations (Murty et al.,…”
This looks like that: deep learning for interpretable image recognition (2018, methodology): “, [50, 49, 3, 24, 15]); other attention models generate attention with “blackbox” methods – e.”
Fine-Grained Image Analysis With Deep Learning: A Survey (2021, unknown): “PS-CNN [68] and Mask-CNN [31] employed segmentation models to get part/object masks to aid part/object localization.”
Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition (2017, background): “A large amount of works propose to leverage the extra annotations of bounding boxes and parts to localize significant regions in fine-grained recognition [9, 16, 22, 30, 33, 34].”
Mask-CNN: Localizing parts and selecting descriptors for fine-grained bird species categorization (2018, methodology): “Among them, the part-stacked CNN model [7] is the most related work to ours.”
Dynamic Computational Time for Visual Attention (2017, background): “Fine-grained image recognition has been extensively studied in recent years [7, 6, 15, 30, 37, 39, 35, 44, 50].”
DeepVoting: An Explainable Framework for Semantic Part Detection under Partial Occlusion (2017, methodology): “…can directly assist object detection (Felzenszwalb et al. 2010; Zhu, Chen, and Yuille 2015), be used for a ﬁne-scaled description of the object (Zhang et al. 2014), or serve as auxiliary cues to understand classiﬁcation (Huang et al. 2016), object detection or segmentation (Wang et al. 2015b).”
Body Joint Guided 3-D Deep Convolutional Descriptors for Action Recognition (2017, background): “[23] and [24] took an extra step to train one of the two CNNs in bilinear model explicitly with part annotations for keypoints prediction.”
Fully Convolutional Attention Networks for Fine-Grained Recognition (2016, background): “Conventional CNN approach (left) ﬁnds difﬁculty in differentiating similar ﬁne-grained categories with subtle local variations (eg., Rhinoceros Auklet against Parakeet Auklet).”
WildFish: A Large Benchmark for Fish Recognition in the Wild (2018, background): “Furthermore, the extra annotations of bounding boxes and parts are often used as a strong supervision to detect distinct regions of fine-grained categories [21, 25, 46, 47].”
Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches (2020, background): “Early work mostly finds discriminative regions with the assistance of manual annotations [2,21,34,37,16].”
Adaptive Triplet Model for Fine-Grained Visual Categorization (2018, background): “Remarkably, ourmodels are comparable to or even better than thosemethods [4], [13]–[15], [20], [21], [27], [39] with extra annotations, including bounding box and part annotation.”
Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding (2018, background): “To better capture subtle visual di erence among sub-ordinate categories, a series works [16, 46, 47] were also proposed to leverage extra supervision of bounding boxes and parts to locate discriminative regions.”
Learning a Discriminative Filter Bank Within a CNN for Fine-Grained Recognition (2016, background): “As long as the layer is properly initialized, the whole network can be trained in an end-to-end fashion just once, which is more efficient compared with the multistage training strategy of previous works [29, 50, 18].”
Low-Rank Bilinear Pooling for Fine-Grained Classification (2016, background): “One approach to dealing with such nuisance parameters has been to exploit strong supervision, such as detailed part-level, keypoint-level and attribute annotations [38, 9, 36].”
Learning Rich Part Hierarchies With Progressive Attention Networks for Fine-Grained Image Recognition (2020, background): “Most part-based methods focus on learning explicit part localizations, which involves either supervised part detection technique [6], [12], [18], [24]–[26] or unsupervised/weakly supervised attention learning [13], [14], [20], [21], [27], [28].”
A survey on deep learning-based fine-grained object classification and semantic segmentation (2017, methodology): “Part-based R-CNN [19] AlexNet BBox + Parts BBox 76.4 Part-based R-CNN [19] AlexNet BBox + Parts − 73.9 Multi-proposal consensus [22] AlexNet BBox BBox 80.3 PoseNorm [24] Alexnet BBox + Parts − 75.7 PS-CNN [26] AlexNet BBox + Parts BBox 76.2 Deep LAC [28] AlexNet BBox BBox 80.3”
Destruction and Construction Learning for Fine-Grained Image Recognition (2019, methodology): “These two-steps methods [21, 11, 1] mostly need additional bounding box annotations on objects or parts, which are expensive to collect.”