Bilinear CNN Models for Fine-Grained Visual Recognition

By Tsung-Yu Lin, Aruni RoyChowdhury, Subhransu Maji. 2015

2,076 citations recorded by the source corpus.

We propose bilinear models, a recognition architecture that consists of two feature extractors whose outputs are multiplied using outer product at each location of the image and pooled to obtain an image descriptor. This architecture can model local pairwise feature interactions in a translationally invariant manner which is particularly useful for fine-grained categorization. It also generalizes various orderless texture descriptors such as the Fisher vector, VLAD and O2P. We present experiments with bilinear models where the feature extractors are based on convolutional neural networks. The bilinear form simplifies gradient computation and allows end-to-end training of both networks using image labels only. Using networks initialized from the ImageNet dataset followed by domain specific fine-tuning we obtain 84.1% accuracy of the CUB-200-2011 dataset requiring only category labels at training time. We present experiments and visualizations that analyze the effects of fine-tuning and the choice two networks on the speed and accuracy of the models. Results show that the architecture compares favorably to the existing state of the art on a number of fine-grained datasets while being substantially simpler and easier to train. Moreover, our most accurate model is fairly efficient running at 8 frames/sec on a NVIDIA Tesla K40 GPU. The source code for the complete system will be made available at http://vis-www.cs.umass.edu/bcnn.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Do Better ImageNet Models Transfer Better? (2018, methodology): “A substantial body of existing research indicates that, in image tasks, fine-tuning typically achieves higher accuracy than classification based on fixed features, especially for larger datasets or datasets with a larger domain mismatch from the training set [1, 6, 24, 74, 2, 43, 33, 9, 51].”
ALICE: Active Learning with Contrastive Natural Language Explanations (2020, positive): “Following Lin et al. (2015); Hu and Qi (2019), we view the activation map φ(x) ∈ RH×W×d of each image x as H×W attention keys k1, ..., kH×W ∈ Rd.”
Fine-Grained Image Classification via Combining Vision and Language (2017, neutral): “Compared with the meth-ods [37, 38, 19, 2, 5] which do not use object and part annotations, our CVL approach obtains a 1.01% higher accuracy than the best performing result of PD [5].”
Spatial Transformer Networks (2015, methodology): “In Table 1 (right), we show the transformations performed for some test cases where a CNN is unable to correctly classify the digit, but a spatial transformer network can.”
Learning feed-forward one-shot learners (2016, methodology): “A variant is to combine embeddings using the outer-product, which yields a bilinear classification rule [13].”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, methodology): “Bilinear pooling models (Tenenbaum and Freeman, 2000) have recently been shown to be beneficial for finegrained classification for vision only tasks (Lin et al., 2015).”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “This includes simple linear models using ranking loss [16, 36], bi-linear pooling methods [45, 20, 55, 17, 37], attention-based methods [1, 52, 64] and reasoning based approaches [54, 27, 33, 38, 29] on CLEVER and VQA-v1.”
MUTAN: Multimodal Tucker Fusion for Visual Question Answering (2017, background): “Bilinear interactions have shown great success in deep learning for fine-grained classification [16], and Multimodal language modeling [10].”
Recent advances in convolutional neural networks (2015, background): “[112] proposes bilinear models for fine-grained image classification.”
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, methodology): “Since fine-grained recognition is a task inherently difficult for humans, this typically requires either recruiting a team of experts [58,38] or extensive crowd-sourcing pipelines [30,4].”
This looks like that: deep learning for interpretable image recognition (2018, background): “4%, which is on par with some state-of-the-art models on this dataset, such as B-CNN [25] (91.”
Convolutional Two-Stream Network Fusion for Video Action Recognition (2016, methodology): “To make bilinear features usable in practice, it is usually applied at ReLU5, the fully-connected layers are removed [15] and power-and L 2 -normalisation is applied for effective classi-ﬁcation with linear SVMs.”
ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering (2015, methodology): “Attention models: Attention models have been successfully adopted in many computer vision tasks, including object detection [20, 2], fine-grained image classification [22, 15], and image captioning [9].”
Deep Image: Scaling up Image Recognition (2015, methodology): “Species categorization results on aircraft variant Method Accuracy(%) B-CNN(M,M), w/ ft [30] 73.”
A2-Nets: Double Attention Networks (2018, methodology): “A recent work [15] used bilinear pooling to capture second-order statistics of features and generate global representations.”
Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering (2017, methodology): “In contrast to linear pooling, bilinear pooling [11] has recently been used to integrate different CNN features for fine-grained image recognition [12].”
Visual question answering: Datasets, algorithms, and future challenges (2016, background): “Similar ideas were shown to work well for improving fine-grained image recognition [71].”