Neural Activation Constellations: Unsupervised Part Model Discovery with Convolutional Networks

By Marcel Simon, Erik Rodner. 2015

426 citations recorded by the source corpus.

Part models of object categories are essential for challenging recognition tasks, where differences in categories are subtle and only reflected in appearances of small parts of the object. We present an approach that is able to learn part models in a completely unsupervised manner, without part annotations and even without given bounding boxes during learning. The key idea is to find constellations of neural activation patterns computed using convolutional neural networks. In our experiments, we outperform existing approaches for fine-grained recognition on the CUB200-2011, Oxford PETS, and Oxford Flowers dataset in case no part or bounding box annotations are available and achieve state-of-the-art performance for the Stanford Dog dataset. We also show the benefits of neural constellation models as a data augmentation technique for fine-tuning. Furthermore, our paper unites the areas of generic and fine-grained classification, since our approach is suitable for both scenarios.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Spatial Transformer Networks (2015, methodology): “In Table 1 (right), we show the transformations performed for some test cases where a CNN is unable to correctly classify the digit, but a spatial transformer network can.”
Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, background): “discovered objects from feature maps of unlabeled images [28].”
Interpreting CNNs via Decision Trees (2018, methodology): “Theoretically, our method also supports other techniques of mining interpretable features in middle layers [38, 27].”
Interpretable Convolutional Neural Networks (2017, background): “discovered objects from feature maps of unlabeled images [22], and selected a certain filter to describe each semantic part in a supervised fashion [23].”
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, background): “The majority of research in fine-grained recognition has focused on developing improved models for classification [2, 5, 7, 15, 17, 18, 24, 30, 37, 44, 45, 58, 59, 60, 63, 64].”
This looks like that: deep learning for interpretable image recognition (2018, neutral): “Examples of attention models include class activation maps [56] and various part-based models (e.g., [55, 53, 15, 57, 43, 10, 9, 34, 37, 49, 7]; see Table 1).”
Bilinear CNN Models for Fine-Grained Visual Recognition (2015, methodology): “a ﬁne-tuned VGG-D network that obtains 63.7% accuracy. Van Horn et al. [64] obtains 75% accuracy using AlexNet and part annotations at test time, while the “neural activation constellations” approach [57] obtains 76.3%accuracy using a GoogLeNet architecture [60]. Comparison to other techniques. Tabl”
Deep Image: Scaling up Image Recognition (2015, methodology): “Table 3. Species categorization results on Oxford flowers Method Accuracy(%) Angelova et al. [1] 80.66 Murray et al. [34] 84.60 Razavian et al. [40] 86.80 Azizpour et al. [2] 91.30 No parts (AlexNet) [44] 90.35 Const. (AlexNet) [44] 91.74 No parts (VGG19) [44] 93.07 Const. (VGG19) [44] 95.34 Ours w/”
The Devil is in the Tails: Fine-grained Classification in the Wild (2017, background): “Most ﬁne-grained research papers present a novel model for classiﬁcation [4, 6, 7, 12, 15, 17, 25, 31, 40, 45, 46, 57, 58, 59, 61, 62].”
Examining CNN representations with respect to Dataset Bias (2017, methodology): “Zhou et al. (Zhou et al. 2015; 2016) mined patterns for “scene” semantics from feature maps of a CNN. Simon et al. discovered objects (Simon and Rodner 2015) from CNN feature maps in an unsupervised manner, and retrieved patterns for object parts in a supervised fashion (Simon, Rodner, and Denzler…”
Interpreting CNN knowledge via an Explanatory Graph (2017, background): “discovered objects from feature maps of unlabeled images [16], and selected a filter to describe each part in a supervised fashion [17].”
Unsupervised learning of object semantic parts from internal states of CNNs by population encoding (2015, methodology): “[19] and [14] used activations from intermediate layers of the CNN to ﬁnd object parts, which they then used for ﬁne-grained object classiﬁcation.”
This Looks Like That: Deep Learning for Interpretable Image Recognition (2018, background): “[29, 38]), or propose a number of candidate parts using selective search-based region proposal network [34, 11, 10, 26] or Monte Carlo sampling [18].”
Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition (2017, methodology): “The framework are typically composed of two independent steps: 1) part localization by training from positive/nagetive image patches [35] or pinpointing from pre-trained feature channels [25], and 2) fine-grained feature learning by selective pooling [31] or dense encoding from feature maps [17].”
Learning Rich Part Hierarchies With Progressive Attention Networks for Fine-Grained Image Recognition (2020, methodology): “, stripe) [13], [19], it is usually difficult to express rich and robust part information by a single channel.”
Adaptive Triplet Model for Fine-Grained Visual Categorization (2018, background): “NAC [17] is another unsupervised part discovery model, which trains part selectors by finding constellations of activation patterns.”
Audio Visual Attribute Discovery for Fine-Grained Object Recognition (2018, methodology): “Even we also have a superiority on the accuracy comparing with (Simon and Rodner 2015), which uses the part annotation to learn the classifiers.”