Low-Rank Bilinear Pooling for Fine-Grained Classification

By Shu Kong, Charless C. Fowlkes. Computer Vision and Pattern Recognition, 2016

375 citations recorded by the source corpus.

Pooling second-order local feature statistics to form a high-dimensional bilinear feature has been shown to achieve state-of-the-art performance on a variety of fine-grained classification tasks. To address the computational demands of high feature dimensionality, we propose to represent the covariance features as a matrix and apply a low-rank bilinear classifier. The resulting classifier can be evaluated without explicitly computing the bilinear feature map which allows for a large reduction in the compute time as well as decreasing the effective number of parameters to be learned. To further compress the model, we propose a classifier co-decomposition that factorizes the collection of bilinear classifiers into a common factor and compact per-class terms. The co-decomposition idea can be deployed through two convolutional layers and trained in an end-to-end architecture. We suggest a simple yet effective initialization that avoids explicitly first training and factorizing the larger bilinear classifiers. Through extensive experiments, we show that our model achieves state-of-the-art performance on several public datasets for fine-grained classification trained with only category labels. Importantly, our final model is an order of magnitude smaller than the recently proposed compact bilinear model [8], and three orders smaller than the standard bilinear CNN model [19].

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f24bbac95cc735c42559d53a33eaf5d689e316c1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

On Modality Bias in the TVQA Dataset (2020, neutral): “Joint representations (e.g. concatenation, bilinear pooling [8, 16, 17, 28, 30]) combine unimodal signals into the same representation space.”
Multi-view Harmonized Bilinear Network for 3D Object Recognition (2018, methodology): “Recently it is used in pooling local convolutional features [20, 7, 5, 17] to take the second-order statistics into consideration, achieving state-of-the-art performance in fine-grained image classification.”
Fully Convolutional Attention Networks for Fine-Grained Recognition (2016, background): “Most of the current state-ofthe-art fine-grained recognition algorithms are also based on deep CNN representation to distinguish the subtle difference [23, 24, 25].”
From BoW to CNN: Two Decades of Texture Representation for Texture Classification (2018, background): “In [100], Kong and Fowlkes proposed to represent the bilinear features as a matrix and applied a low rank bilinear classifier.”
Dynamic Computational Time for Visual Attention (2017, methodology): “We observe that methods including Bilinear CNN [54, 53] (84.1% - 84.2%), Spatial Transformer Networks [59] (84.1”
Recurrent Pixel Embedding for Instance Grouping (2017, methodology): “To compute calibrated cosine similarity, we utilize an L2-normalization layer before matrix multiplication [44], which also contains random sampling with a hyper-parameter to control the ratio of pixels to be sampled for an image.”
Attentional Pooling for Action Recognition (2017, positive): “Our work is most related to [26], who point out when efﬁciently implemented, low-rank approximations avoid explicitly computing second-order features.”
Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding (2018, neutral): “[19] proposed classi er co-decomposition to further compress the bilinear model.”
Anticipative Video Transformer (2021, background): “The general idea of self-attention in vision dates back to non-local means [9], and is incorporated into contemporary network architectures as non-local blocks [10, 54, 90, 92] and gating mechanisms [30, 45, 60, 94].”
FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery (2018, background): “, different species of birds), which requires annotations from domain experts [36, 53, 34, 4, 13, 8, 33, 12, 28, 59, 47].”
Multi-scale Location-Aware Kernel Representation for Object Detection (2018, background): “The recent researches on challenging fine-grained visual categorization [4, 22, 38, 6] show that high-order statistics representations can capture more discriminative information than first-order ones, and obtain promising improvements.”
Destruction and Construction Learning for Fine-Grained Image Recognition (2019, methodology): “This method was later extended to a series of related works with further improvements [12, 4, 8].”
Grassmann Pooling as Compact Homogeneous Bilinear Pooling for Fine-Grained Visual Classification (2018, methodology): “Similar to previous methods, the bilinear pooling method and its variants [21, 23, 24] also find that proper feature normalization provides a non-trivial improvement on performance.”