Compact Bilinear Pooling

By Yang Gao, Oscar Beijbom, Ning Zhang, Trevor Darrell. 2015

859 citations recorded by the source corpus.

Bilinear models has been shown to achieve impressive performance on a wide range of visual tasks, such as semantic segmentation, fine grained recognition and face recognition. However, bilinear features are high dimensional, typically on the order of hundreds of thousands to a few million, which makes them impractical for subsequent analysis. We propose two compact bilinear representations with the same discriminative power as the full bilinear representation but with only a few thousand dimensions. Our compact representations allow back-propagation of classification errors enabling an end-to-end optimization of the visual recognition system. The compact bilinear representations are derived through a novel kernelized analysis of bilinear pooling which provide insights into the discriminative power of bilinear pooling, and a platform for further research in compact pooling methods. Experimentation illustrate the utility of the proposed representations for image classification and few-shot learning across several datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Do Better ImageNet Models Transfer Better? (2018, methodology): “Several papers have proposed methods to make better use of CNN features and thus improve the efficacy of transfer learning [43, 11, 42, 23, 73, 63, 13, 41, 54].”
Attention on Attention for Image Captioning (2019, methodology): “AoA generates an attention gate and an information vector via two linear transformations and applies the gate to the vector to add a second attention, where the techniques are similar to some other work: GLU [10], which replaces RNN and CNN to capture long-range dependencies for language modeling; m”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, methodology): “In this paper, we adopt the idea from Gao et al. (2016) which shows how to efﬁciently compress bilinear pooling for a single modality.”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “This includes simple linear models using ranking loss [16, 36], bi-linear pooling meth-ods [45, 20, 55, 17, 37], attention-based methods [1, 52, 64] and reasoning based approaches [54, 27, 33, 38, 29] on CLEVER and VQA-v1.”
Inferring and Executing Programs for Visual Reasoning (2017, background): “Many systems for VQA employ a very similar architecture [3, 8, 9, 30, 32, 33, 44]: they combine an RNN-based embedding of the question with a convolutional network-based embedding of an image in a classification model over possible answers.”
Generating Visual Explanations (2016, methodology): “Second, we use rich category speciﬁc features [3] to generate relevant explanations.”
SplitNN-driven Vertical Partitioning (2020, methodology): “One can readily employ other encoding methods like Compact Bilinear Pooling (Diba et al. , 2017; Gao et al. , 2016), Temporal Compact Bilinear Pooling (Sharma et al. , 2020), NetVLAD (Arandjelovi ´ c et al. , 2016) instead of the pooling mechanisms for a more robust representation learning.”
Stacking with Auxiliary Features for Visual Question Answering (2018, methodology): “The question and image features are high-dimensional and therefore a neural network classiﬁer worked (Andreas et al., 2016b) 58.70 81.20 37.70 44.00 LSTM (Antol et al., 2015) 58.20 80.60 36.50 43.70 HieCoAtt (Lu et al., 2016) 61.80 79.70 38.70 51.70 MCB (Single system) (Fukui et al., 2016) well.”
Bilinear CNN Models for Fine-Grained Visual Recognition (2015, methodology): “s to obtain 83.6% accuracy, and 84.5% accuracy with FC and FV pooling. Since its publication, the B-CNN model has been improved by others in several ways. First, the compact bilinear pooling approach [19] was proposed to reduce the size of the bilinear features (we evaluate this in Section5.1). Zhan”
DSSD : Deconvolutional Single Shot Detector (2017, methodology): “We also try to use the approximate bilinear pooling method[9], the low-dimensional approximation of the original method proposed by Lin et al. [17], but training speed is slowing down and the training error decrease very slowly as well.”
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering (2018, methodology): “To capture the high-level interactions between the two modalities, Bilinear Fusion [9] has been proposed to adopt bilinear pooling to fuse features from the two modalities.”
Deep Layer Aggregation (2017, background): “The numbers of Baseline, Compact [13] and Kernel [9] are directly cited from [9].”
Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering (2020, background): “For Image QA, there have been extensive efforts concentrated on (Yang et al. 2016; Xu and Saenko 2016; Wu et al. 2016; Kazemi and Elqursh 2017; Anderson et al. 2018; Gao et al. 2016; Fukui et al. 2016; Ben-Younes et al. 2017) in the last decade.”
Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning (2018, background): “This idea was later extended to compact bilinear pooling [17], and then higher order interactions [11, 9, 47].”
Attentive Explanations: Justifying Decisions and Pointing to the Evidence (2016, methodology): “Instead of doing Compact Bilinear Pooling [16] between the two representations, our model simply embeds the encoded image feature using 1 × 1 convolutions and applies element-wise multiplication between the image embedding and the LSTM feature.”
Learnable pooling with Context Gating for video classification (2017, methodology): “To alleviate this issue, we tried to reduce the output dimension using the multi-modal compact bilinear pooling approach [52] but found the resulting models underfitting the data.”
Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding (2018, methodology): “luding FGVC-Aircraft [29], Stanford Cars [21], Caltech-UCSD birds [39], Stanford Dogs [18], Oxford Flowers [31], to name a few. As a representative dataset that was widely used in previous FGIR works [9, 12, 27], Caltech-UCSD birds dataset contained 11,788 images and covered 200 species of birds. Th”
Bilinear CNNs for Fine-grained Visual Recognition (2015, methodology): “CBP approximates the outer product using a product of sparse linear projections of features with a Tensor Sketch [48].”