Regularizing Neural Networks via Minimizing Hyperspherical Energy

By Rongmei Lin, Weiyang Liu, Zhen Liu, Zhiding Yu, James M. Rehg, Li Xiong, Le Song. 2019

48 citations recorded by the source corpus.

Inspired by the Thomson problem in physics where the distribution of multiple propelling electrons on a unit sphere can be modeled via minimizing some potential energy, hyperspherical energy minimization has demonstrated its potential in regularizing neural networks and improving their generalization power. In this paper, we first study the important role that hyperspherical energy plays in neural network training by analyzing its training dynamics. Then we show that naively minimizing hyperspherical energy suffers from some difficulties due to highly non-linear and non-convex optimization as the space dimensionality becomes higher, therefore limiting the potential to further improve the generalization. To address these problems, we propose the compressive minimum hyperspherical energy (CoMHE) as a more effective regularization for neural networks. Specifically, CoMHE utilizes projection mappings to reduce the dimensionality of neurons and minimizes their hyperspherical energy. According to different designs for the projection mapping, we propose several distinct yet well-performing variants and provide some theoretical guarantees to justify their effectiveness. Our experiments show that CoMHE consistently outperforms existing regularization methods, and can be easily applied to different neural networks.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Weiyang Liu
2. Adrian Weller
3. Zhen Liu
4. Rongmei Lin
5. Lanqing Hong
6. Tao Mei
7. Bhiksha Raj
8. Zhen Lei

In Computational Mechanics

1. Rongmei Lin
2. Tao Mei
3. Adrian Weller
4. Zhen Lei
5. Rita Singh
6. Bhiksha Raj
7. Bernhard Schölkopf
8. Li Xiong

In Computer Vision and Pattern Recognition

1. Xiaogang Wang
2. Luc Van Gool
3. Dacheng Tao
4. Yi Yang
5. Yu Qiao
6. Weiyang Liu
7. Andrew Zisserman
8. Chen Change Loy

Citations in the corpus

• Continual Learning by Modeling Intra-Class Variation (2022, background): “Second, the representation space is more regularized, since it converges to the simplex equiangular tight frame (Papyan et al., 2020) which is also equivalent to a hyperspherically uniform space (Liu et al., 2018a; Lin et al., 2020; Liu et al., 2021c;b).”
• SphereFace Revived: Unifying Hyperspherical Face Recognition (2021, background): “Beyond face recognition, the idea of learning a representation on the hypersphere is also shown generally useful in a diverse set of applications, such as few-shot recognition [47], [48], [49], [50], [51], deep metric learning [25], [52], self-supervised learning [53], [54], [55], [56], generative m”
• Learning with Hyperspherical Uniformity (2021, background): “energy that distributes N electrons on a unit sphere as evenly as possible, MHE [46, 45] encourages hyperspherical diversity and defines the following learning objective for n d-dimensional vectors (e.”
• Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition (2020, background): “For face recognition model compression, there have been several attempts [16, 40, 51, 26, 19, 52, 18, 35, 9, 10] in literatures to distil large CNNs, so as to make their deployments easier.”