Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization

By P. Li, Jiangtao Xie, Qilong Wang, Zilin Gao. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

305 citations recorded by the source corpus.

Global covariance pooling in convolutional neural networks has achieved impressive improvement over the classical first-order pooling. Recent works have shown matrix square root normalization plays a central role in achieving state-of-the-art performance. However, existing methods depend heavily on eigendecomposition (EIG) or singular value decomposition (SVD), suffering from inefficient training due to limited support of EIG and SVD on GPU. Towards addressing this problem, we propose an iterative matrix square root normalization method for fast end-to-end training of global covariance pooling networks. At the core of our method is a meta-layer designed with loop-embedded directed graph structure. The meta-layer consists of three consecutive nonlinear structured layers, which perform pre-normalization, coupled matrix iteration and post-compensation, respectively. Our method is much faster than EIG or SVD based ones, since it involves only matrix multiplications, suitable for parallel implementation on GPU. Moreover, the proposed network with ResNet architecture can converge in much less epochs, further accelerating network training. On large-scale ImageNet, we achieve competitive performance superior to existing counterparts. By finetuning our models pre-trained on ImageNet, we establish state-of-the-art results on three challenging fine-grained benchmarks. The source code and network models will be available at http://www.peihuali.org/iSQRT-COV.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2451db113552afb6d9ad15ef4009ec4133d28f74

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiebo Luo
2. Tao Mei
3. Zhengjun Zha
4. Jianlong Fu
5. Yi-Zhe Song
6. Ling Shao
7. Heliang Zheng
8. Honggang Qi

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ling Shao
4. Jiebo Luo
5. Alan L. Yuille
6. Ming-Hsuan Yang
7. Yu Qiao
8. Yi Yang

Citations in the corpus

• Learning Deep Bilinear Transformation for Fine-grained Image Representation (2019, unknown): “…distinct characteristics existed in fine-grained recognition datasets: 1) well-structured, e.g., different birds share similar semantic parts like head, wings and tail [5–9]), 2) rich details, e.g., sophisticated textures are supposed to be useful to distinguish two similar species [10–13].”
• Learning Rich Part Hierarchies With Progressive Attention Networks for Fine-Grained Image Recognition (2020, background): “WE DO NOT COMPARED WITH THE METHODS USING ADDITIONAL DATA [3] AND STRONG BASE MODEL [44]”
• See Better Before Looking Closer: Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classification (2019, background): “…86.5 PC [4] 86.9 DFL-CNN [29] 87.4 NTS-Net [33] 87.5 MPN-COV [12] 88.7 WS-DAN 89.4 [23] 80.5 ResNet-101 [7] 87.2 Inception-V3 [26] 87.4 B-CNN [16] 84.1 RA-CNN [5] 88.4 PC [4] 89.2 GP-256 [31] 89.8 MA-CNN [38] 89.9 MPN-COV [12] 91.4 NTS-Net [33] 91.4 DFL-CNN [29] 92.0 WS-DAN 93.0 Methods Accuracy(”
• Learning Attentive Pairwise Interaction for Fine-Grained Classification (2020, unknown): “…2016) VGGNet-16 Yes 92.5 PA-CNN (Krause et al. 2015) VGGNet-19 Yes 92.8 FCAN (Liu et al. 2016) ResNet-50 Fourth, API-Net even outperforms the approaches with extra supervision such as localization annotations (Zhang et al. 2014), super-category labels (Chen et al. 2018), or text (Li et al. 2018a).”
• Fine-Grained Image Analysis With Deep Learning: A Survey (2021, methodology): “iSQRT-COV [101] and the improved B-CNN [146] used the Newton-Schulz iteration to approximate matrix square-root”
• Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, methodology): “These methods focus on constructing bilinear [28] or second-order pooling [26] by constraining the power of the covariance matrix and are limited to producing fully-connected activations, while our work provides a generic module that can be ubiquitously built in various neural network frameworks.”
• Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-Grained Image Recognition (2019, methodology): “• iSQRT-COV [18]: Towards faster training of global covariance pooling networks by iterative matrix square root normalization.”
• The Herbarium Challenge 2019 Dataset (2019, methodology): “the ImageNet [14] and iNaturalist Challenge 2018 datasets, with input image dimensions of 448x448 and standard data augmentation. Additional techniques used include deformable convolutions [8], iSQRT [12] and random erasing [23]. 4. Conclusions and Future Work We have developed the Herbarium Challen”
• Distribution Consistency Based Covariance Metric Networks for Few-Shot Learning (2019, methodology): “Later, to accelerate the computation further, (Li et al. 2018) proposed an iterative matrix square-root normalization method for fast end-to-end training.”