Learning with Hyperspherical Uniformity

By Weiyang Liu, Rongmei Lin, Zhen Liu, Li Xiong, Bernhard Schölkopf, Adrian Weller. 2021

50 citations recorded by the source corpus.

Due to the over-parameterization nature, neural networks are a powerful tool for nonlinear function approximation. In order to achieve good generalization on unseen data, a suitable inductive bias is of great importance for neural networks. One of the most straightforward ways is to regularize the neural network with some additional objectives. L2 regularization serves as a standard regularization for neural networks. Despite its popularity, it essentially regularizes one dimension of the individual neuron, which is not strong enough to control the capacity of highly over-parameterized neural networks. Motivated by this, hyperspherical uniformity is proposed as a novel family of relational regularizations that impact the interaction among neurons. We consider several geometrically distinct ways to achieve hyperspherical uniformity. The effectiveness of hyperspherical uniformity is justified by theoretical insights and empirical evaluations.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Weiyang Liu
2. Zhen Liu
3. Adrian Weller
4. Rongmei Lin
5. Le Song
6. Li Xiong
7. James M. Rehg
8. Lanqing Hong

In Computational Mechanics

1. Rongmei Lin
2. James M. Rehg
3. Adrian Weller
4. Rita Singh
5. Bhiksha Raj
6. Le Song
7. Li Xiong
8. Zhen Liu

Citations in the corpus

• Continual Learning by Modeling Intra-Class Variation (2022, background): “Second, the representation space is more regularized, since it converges to the simplex equiangular tight frame (Papyan et al., 2020) which is also equivalent to a hyperspherically uniform space (Liu et al., 2018a; Lin et al., 2020; Liu et al., 2021c;b).”
• SphereFace Revived: Unifying Hyperspherical Face Recognition (2021, background): “Beyond face recognition, the idea of learning a representation on the hypersphere is also shown generally useful in a diverse set of applications, such as few-shot recognition [47], [48], [49], [50], [51], deep metric learning [25], [52], self-supervised learning [53], [54], [55], [56], generative m”
• Orthogonal Over-Parameterized Training (2020, background): “More different ways to minimize the hyperspherical energy can also be considered [50].”