Orthogonal Over-Parameterized Training

By Weiyang Liu, Rongmei Lin, Zhen Liu, James M. Rehg, Li Xiong, Le Song. 2020

55 citations recorded by the source corpus.

The inductive bias of a neural network is largely determined by the architecture and the training algorithm. To achieve good generalization, how to effectively train a neural network is of great importance. We propose a novel orthogonal over-parameterized training (OPT) framework that can provably minimize the hyperspherical energy which characterizes the diversity of neurons on a hypersphere. By maintaining the minimum hyperspherical energy during training, OPT can greatly improve the empirical generalization. Specifically, OPT fixes the randomly initialized weights of the neurons and learns an orthogonal transformation that applies to these neurons. We consider multiple ways to learn such an orthogonal transformation, including unrolling orthogonalization algorithms, applying orthogonal parameterization, and designing orthogonality-preserving gradient descent. For better scalability, we propose the stochastic OPT which performs orthogonal transformation stochastically for partial dimensions of neurons. Interestingly, OPT reveals that learning a proper coordinate system for neurons is crucial to generalization. We provide some insights on why OPT yields better generalization. Extensive experiments validate the superiority of OPT over the standard training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Weiyang Liu
2. Adrian Weller
3. Zhen Liu
4. Rongmei Lin
5. Lanqing Hong
6. Bhiksha Raj
7. Li Xiong
8. L. Yu

In Computer Vision and Pattern Recognition

1. Weiyang Liu
2. Adrian Weller
3. Zhen Liu
4. Rongmei Lin
5. Bhiksha Raj
6. Lanqing Hong
7. Rita Singh
8. Yandong Wen

Citations in the corpus

• Continual Learning by Modeling Intra-Class Variation (2022, background): “Second, the representation space is more regularized, since it converges to the simplex equiangular tight frame (Papyan et al., 2020) which is also equivalent to a hyperspherically uniform space (Liu et al., 2018a; Lin et al., 2020; Liu et al., 2021c;b).”
• SphereFace Revived: Unifying Hyperspherical Face Recognition (2021, background): “Beyond face recognition, the idea of learning a representation on the hypersphere is also shown generally useful in a diverse set of applications, such as few-shot recognition [47], [48], [49], [50], [51], deep metric learning [25], [52], self-supervised learning [53], [54], [55], [56], generative m”
• Learning with Hyperspherical Uniformity (2021, background): “[51, 52, 49, 48, 50, 80, 79, 20, 19, 60, 14, 47] propose to learn representations on hypersphere and show that angular information in neural networks, in contrast to magnitude, preserves the key semantics and is very crucial to generalization.”
• Provable Lifelong Learning of Representations (2021, background): “…regularization term [Li and Hoiem, 2017, Kirkpatrick et al., 2017], freezing the network from previous tasks and adding branches to new tasks [Xu and Zhu, 2018, Rusu et al., 2016, Liu et al., 2019, Liu et al., 2021, Yoon et al., 2017] and replaying previous tasks’ exemplars [Rebuffi et al., 2017].”