Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet

By Wieland Brendel, Matthias Bethge. 2019

641 citations recorded by the source corpus.

Deep Neural Networks (DNNs) excel on many complex perceptual tasks but it has proven notoriously difficult to understand how they reach their decisions. We here introduce a high-performance DNN architecture on ImageNet whose decisions are considerably easier to explain. Our model, a simple variant of the ResNet-50 architecture called BagNet, classifies an image based on the occurrences of small local image features without taking into account their spatial ordering. This strategy is closely related to the bag-of-feature (BoF) models popular before the onset of deep learning and reaches a surprisingly high accuracy on ImageNet (87.6% top-5 for 33 x 33 px features and Alexnet performance for 17 x 17 px features). The constraint on local features makes it straight-forward to analyse how exactly each part of the image influences the classification. Furthermore, the BagNets behave similar to state-of-the art deep neural networks such as VGG-16, ResNet-152 or DenseNet-169 in terms of feature sensitivity, error distribution and interactions between image parts. This suggests that the improvements of DNNs over previous bag-of-feature classifiers in the last few years is mostly achieved by better fine-tuning rather than by qualitatively different decision strategies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Industrial and Manufacturing Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Industrial and Manufacturing Engineering

Citations in the corpus

Shortcut learning in deep neural networks (2020, methodology): “ying ability one is actually interested in. For example, the ImageNet dataset [75] was intended to measure the ability “object recognition”, but DNNs seem to rely mostly on “counting texture patches” [36]. Likewise, instead of performing “natural language inference”, some language models perform wel”
Natural Adversarial Examples (2019, background): “, 2018 [7] show that classifiers that do not know the spatial ordering of image regions can be competitive on the”
On the Binding Problem in Artificial Neural Networks (2020, background): “However, it has repeatedly been shown that such features are best understood as “a texture detector highly correlated with an object” (Olah et al., 2020; Sundararajan et al., 2017; Ancona et al., 2017; Brendel and Bethge, 2019; Geirhos et al., 2019).”
Invariant Risk Minimization (2019, background): “Unfortunately, spurious correlations and biases are often simpler to detect than the true phenomenon of interest [17, 9, 10, 11].”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, background): “Our choice is supported by BagNet [8] which found that “even complex perceptual tasks like ImageNet can be solved just based on small image features and without any notion of spatial relationships”.”
Transformer in Transformer (2021, background): “visual transformers ignore the local relation and structure information inside the patch which is important for visual recognition [22, 3].”
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation (2019, background): “However, valuable high-frequency information [4] is being thrown out at low layers, which could explain the lower test accuracy.”
Robustness properties of Facebook's ResNeXt WSL models (2019, methodology): “…the statistical regularities enabling high classification performance in Instagram are similar to those observed in ImageNet and standard image recognition models have an inductive bias for exploiting local textural regularities over more global shape-based regularities (Brendel and Bethge, 2019).”
The Intrinsic Dimension of Images and Its Impact on Learning (2021, background): “Another line of work in deep learning has found that neural networks rely heavily on textures which are low-dimensional (Geirhos et al., 2018; Brendel & Bethge, 2019).”
Deep learning classifiers for hyperspectral imaging: A review (2019, methodology): “Also, the BagNet (Brendel and Bethge, 2019) employs a visual bag-of-local-features model to perform the classification, extracting features that are easy to identify and interpret.”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, positive): “An analogous phenomenon has been noticed when training neural networks [e.g. 85, 109, 54, 39, 17, 14, 37, 51, 107, 76, 48, 19, 61, 77].”
Deep Learning Meets Hyperspectral Image Analysis: A Multidisciplinary Review (2019, background): “For example, Neural Networks (NN) can approximate what a traditional bag-of-local-features does with convolutional ﬁlters [19] very well and SVM can be seen as a single layer NN with a hinge loss.”
Reducing Domain Gap by Reducing Style Bias (2021, background): “Some studies have also shown that CNNs perform well when only local textures are given while global shape structures are missing [12, 6], but work poorly in the reverse scenario [3, 14].”
CAT: Cross Attention in Vision Transformer (2021, background): “Local attention in Transformer-based network The relationship between internal information of the patch is vital in vision[52, 53].”
Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks? (2020, background): “Interestingly, ImageNet classifiers can obtain decent accuracy by looking at only a small patch instead of the entire image (Brendel and Bethge, 2019).”
Shape or Texture: Understanding Discriminative Features in CNNs (2021, background): “The results of this comparison are presented in Table 1(a) where both the ResNet50 and the BagNet variants are trained on IN. Note that ‘Stage’ refers to a residual block in a ResNet (i.e., there are ﬁve stages in ResNet) and all experiments in Table 1 use the stage-5 features.”