Deep Hyperspherical Learning

By Weiyang Liu, Yanming Zhang, Xingguo Li, Zhen Liu, Bo Dai, Tuo Zhao, Le Song. 2017

153 citations recorded by the source corpus.

Convolution as inner product has been the founding basis of convolutional neural networks (CNNs) and the key to end-to-end visual representation learning. Benefiting from deeper architectures, recent CNNs have demonstrated increasingly strong representation abilities. Despite such improvement, the increased depth and larger parameter space have also led to challenges in properly training a network. In light of such challenges, we propose hyperspherical convolution (SphereConv), a novel learning framework that gives angular representations on hyperspheres. We introduce SphereNet, deep hyperspherical convolution networks that are distinct from conventional inner product based convolutional networks. In particular, SphereNet adopts SphereConv as its basic convolution operator and is supervised by generalized angular softmax loss - a natural loss formulation under SphereConv. We show that SphereNet can effectively encode discriminative representation and alleviate training difficulty, leading to easier optimization, faster convergence and comparable (even better) classification accuracy over convolutional counterparts. We also provide some theoretical insights for the advantages of learning on hyperspheres. In addition, we introduce the learnable SphereConv, i.e., a natural improvement over prefixed SphereConv, and SphereNorm, i.e., hyperspherical learning as a normalization method. Experiments have verified our conclusions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Learning by Modeling Intra-Class Variation (2022, methodology): “To enable an effective modeling of ∆f , we draw inspirations from the observations in (Liu et al., 2017b; 2018b;a; Chen et al., 2020) that geodesic distance on hypersphere is well aligned with perceptual difference, and propose to model the distribution ∆f on the hypersphere in MOCA.”
Hyperspherical Variational Auto-Encoders (2018, methodology): “Additionally, modeling data by restricting it to a hypersphere provides some natural regularizing properties as noted in (Liu et al., 2017).”
Regularizing Neural Networks via Minimizing Hyperspherical Energy (2019, methodology): “For ImageNet-2012, we use the same data augmentation in [32].”
Decoupled Networks (2018, methodology): “As a non-trivial generalization of [13], our decoupled network is a more generic and unified framework to model both intra-class variation and semantic difference, providing the flexibility to design or learn both magnitude function h(·) and angular function g(·).”
Deep Face Recognition: A Survey (2018, methodology): “Some papers [84], [108] first normalized the weights only and then added angular/cosine margin into loss functions to make the learned features be discriminative.”
Additive Margin Softmax for Face Verification (2018, methodology): “Then we scale the cosine values using a hyper-parameter s as suggested in [19, 11, 12].”
Orthogonal Projection Loss (2021, unknown): “Orthogonality of kernels in DNNs is well explored with an aim to diversify the learned weight vectors [34, 56].”
SphereFace Revived: Unifying Hyperspherical Face Recognition (2021, methodology): “We can simply use a linear function as the target and non-target functions, as proposed in [31].”
Efficient Low-Resolution Face Recognition via Bridge Distillation (2020, background): “Recent face recognition models proposed designing effective loss functions [5], [23], [28]–[31] for feature learning, that can facilitate the hallucination-based approaches.”
Orthogonal Over-Parameterized Training (2020, methodology): “[4, 54, 7, 26, 78] use orthogonality as a regularization for neurons.”
Angular Visual Hardness (2019, background): “[40] proposes a hyperspherical neural network that constrains the parameters of neurons on a unit hypersphere and uses angular similarity to replace the inner product similarity.”
Hyperspherical Prototype Networks (2019, unknown): “The potential of angular similarities has also been investigated in other layers of deep networks [24, 25].”
Learning towards Minimum Hyperspherical Energy (2018, methodology): “Methods other than diversity-promoting regularization have been widely proposed to improve CNNs [44, 20, 33, 30] and generative adversarial nets (GANs) [4, 34].”
ArcFace: Additive Angular Margin Loss for Deep Face Recognition (2018, unknown): “tremely large when the feature norm from low-quality face image is very small, which potentially increases the risk of gradient explosion. The advantages of feature normalisation are also revealed in [25,26,43,45] and the feature normalisation is explained from analytic, geometric and experimental p”
Learning with Hyperspherical Uniformity (2021, background): “A popular choice out of many existing relational regularizations is the orthogonality regularization [13, 52, 4, 32, 33, 63].”
Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition (2020, methodology): “) [29] or minimum hyperspherical energy (MHE) [27] among filters has been a popular choice for encouraging diversity.”