Is Second-Order Information Helpful for Large-Scale Visual Recognition?

By P. Li, Jiangtao Xie, Qilong Wang, Wangmeng Zuo. 2017

280 citations recorded by the source corpus.

By stacking layers of convolution and nonlinearity, convolutional networks (ConvNets) effectively learn from lowlevel to high-level features and discriminative representations. Since the end goal of large-scale recognition is to delineate complex boundaries of thousands of classes, adequate exploration of feature distributions is important for realizing full potentials of ConvNets. However, state-of-theart works concentrate only on deeper or wider architecture design, while rarely exploring feature statistics higher than first-order. We take a step towards addressing this problem. Our method consists in covariance pooling, instead of the most commonly used first-order pooling, of highlevel convolutional features. The main challenges involved are robust covariance estimation given a small sample of large-dimensional features and usage of the manifold structure of covariance matrices. To address these challenges, we present a Matrix Power Normalized Covariance (MPNCOV) method. We develop forward and backward propagation formulas regarding the nonlinear matrix functions such that MPN-COV can be trained end-to-end. In addition, we analyze both qualitatively and quantitatively its advantage over the well-known Log-Euclidean metric. On the ImageNet 2012 validation set, by combining MPN-COV we achieve over 4%, 3% and 2.5% gains for AlexNet, VGG-M and VGG-16, respectively; integration of MPN-COV into 50-layer ResNet outperforms ResNet-101 and is comparable to ResNet-152. The source code will be available on the project page: http://www.peihuali.org/MPN-COV.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiashi Feng
2. Shuicheng Yan
3. Jiebo Luo
4. Lei Zhang
5. Yunpeng Chen
6. Wangmeng Zuo
7. Xiansheng Hua
8. Jing Huo

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jiashi Feng
3. Lei Zhang
4. Shuicheng Yan
5. Dacheng Tao
6. Ming-Hsuan Yang
7. Wangmeng Zuo
8. Yi Yang

Citations in the corpus

• A2-Nets: Double Attention Networks (2018, methodology): “The double-attention block is related to a number of recent works, including the Squeeze-andExcitation Networks [11], covariance pooling [14], the Non-local Neural Networks [25] and the Transformer architecture of [24].”
• Second-Order Attention Network for Single Image Super-Resolution (2019, background): “On the other hand, recent works [19, 21] have shown that second-order statistics in deep CNNs are more helpful for more discriminative representations than first-order ones.”
• Distribution Consistency Based Covariance Metric Networks for Few-Shot Learning (2019, methodology): “For example, (Li et al. 2017) proposed a Matrix Power Normalized Covariance (MPN-COV) method by producing covariance matrices over the last convolutional features to act as global image representations.”
• Multi-scale Location-Aware Kernel Representation for Object Detection (2018, methodology): “Thereinto, the global second-order pooling methods [18, 25, 28] are plugged into deep CNNs to represent whole images, in which the sum of outer product of convolutional features is firstly computed, then element-wise power normalization [28], matrix logarithm normalization [18] and matrix power…”
• HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation (2019, background): “Especially in the field of fine-grained image/video recognition [24, 8], secondorder statistics such as Covariance and Gaussian descriptors, have demonstrated better performance than descriptors exploiting zeroth- or first-order statistics [22, 24, 40].”