Learning towards Minimum Hyperspherical Energy

By Weiyang Liu, Rongmei Lin, Z. Liu, Lixin Liu, Zhiding Yu, Bo Dai, Le Song. 2018

172 citations recorded by the source corpus.

Neural networks are a powerful class of nonlinear functions that can be trained end-to-end on various applications. While the over-parametrization nature in many neural networks renders the ability to fit complex functions and the strong representation power to handle challenging tasks, it also leads to highly correlated neurons that can hurt the generalization ability and incur unnecessary computation cost. As a result, how to regularize the network to avoid undesired representation redundancy becomes an important issue. To this end, we draw inspiration from a well-known problem in physics -- Thomson problem, where one seeks to find a state that distributes N electrons on a unit sphere as evenly as possible with minimum potential energy. In light of this intuition, we reduce the redundancy regularization problem to generic energy minimization, and propose a minimum hyperspherical energy (MHE) objective as generic regularization for neural networks. We also propose a few novel variants of MHE, and provide some insights from a theoretical point of view. Finally, we apply neural networks with MHE regularization to several challenging tasks. Extensive experiments demonstrate the effectiveness of our intuition, by showing the superior performance with MHE regularization.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Improving Neural Language Modeling via Adversarial Training (2019, methodology): “We also draw theoretical insights on this simple mechanism, showing that it implicitly promotes diversity among the output embedding vectors { w i } , which is widely believed to increase robustness of the results (e.g., Cortes & Vapnik, 1995; Liu et al., 2018b; Gao et al., 2019).”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “Moreover, we empirically observe from Liu et al. (2016; 2018b); Wen et al. (2019) that the intra-class representations, when projected onto a unit hypersphere, are centered around the class mean and distributed like a vMF distribution.”
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (2020, background): “, the Thomson problem of finding the minimal electrostatic potential energy configuration of electrons (Thomson, 1904), and minimization of the Riesz s-potential (Götz & Saff, 2001; Hardin & Saff, 2005; Liu et al., 2018).”
Learning with Hyperspherical Uniformity (2021, methodology): “MHE has been extensively studied and shown to be effective in many applications [46].”
ArcFace: Additive Angular Margin Loss for Deep Face Recognition (2018, methodology): “SphereFace [15] and CosFace [35]) but also outperforming other published methods [36, 14].”
Hyperspherical Prototype Networks (2019, unknown): “(Liu et al., 2018) similarly aims for large margin polar separation of class vectors in the output space by adding the separation as a regularization to a softmax-based deep network.”
Angular Visual Hardness (2019, background): “As a result, it shows the necessity and importance of designing loss functions that focus on optimizing angles, such as [35, 39, 41].”
Efficient Low-Resolution Face Recognition via Bridge Distillation (2020, methodology): “Beyond the base-line softmax loss, we conduct the experiments with four kinds of face recognition losses (CenterLoss [58], SphereFace+ [30], CosFace [5] and ArcFace [31]) on four student models.”
SphereFace Revived: Unifying Hyperspherical Face Recognition (2021, methodology): “Existing hyperspherical FR methods either preserve the feature magnitude in the loss function [1], [2], [32] or normalize the feature magnitude to constant s [9], [30].”
Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning (2019, background): “Convolutional neural networks (CNNs) have become the dominant approach for many computer vision tasks [8, 17, 2, 18, 38, 11, 37].”
Deep Imbalanced Learning for Face Recognition and Attribute Prediction (2018, background): “[37] proposed to maximize the hyperspherical margin regardless of class imbalance, while Wang et al.”
ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks (2019, unknown): “42% Sphereface+ [37] 0.”
Orthogonal Over-Parameterized Training (2020, methodology): “Based on [49], the hyperspherical energy of n neurons is defined as E(v̂i|i=1)= ∑n i=1 ∑n j=1,j 6=i ‖v̂i− v̂j‖ −1 in which v̂i= vi ‖vi‖ is the i-th neuron weight projected onto the unit hypersphere S= {v∈Rd| ‖v‖=1}.”
Regular Polytope Networks (2021, background): “For this purpose, very recently, the works [32], [33] and [34] add a regularization loss to specifically force the classifier weights to be far from each other in a global manner.”
Regularizing Neural Networks via Minimizing Hyperspherical Energy (2019, methodology): “Compared to full-space MHE, the half-space variant [12] further eliminates the collinear redundancy by constructing virtual neurons with the opposite direction to the original ones and then minimizing their hyperspherical energy together.”
Face recognition based on adaptive margin and diversity regularization constraints (2020, background): “The diversity of neurons is promoted by minimizing the hyperspherical energy of neurons on the hypersphere [22, 23].”
Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition (2020, methodology): “To achieve this, several methods [29, 27, 5] have been proposed.”