Centered Weight Normalization in Accelerating Training of Deep Neural Networks

By Lei Huang, Xianglong Liu, Yang Liu, Bo Lang, Dacheng Tao. 2017

83 citations recorded by the source corpus.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Andrew Brock
2. Samuel L. Smith
3. Soham De
4. Li Liu
5. Ling Shao
6. Jia Deng
7. Karen Simonyan
8. Liang Wang

In Computer Vision and Pattern Recognition

1. Ling Shao
2. Luc Van Gool
3. Dacheng Tao
4. Ming-Hsuan Yang
5. Alan L. Yuille
6. Xiaogang Wang
7. Andrew Zisserman
8. Trevor Darrell

In Optimization & Training

1. Soham De
2. Andrew Brock
3. Samuel L. Smith
4. Karen Simonyan
5. Ling Shao
6. Jie Qin
7. Fan Zhu
8. Jia Deng

Citations in the corpus

• High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “(2021) prevent the emergence of a mean-shift in the hidden activations by introducing Scaled Weight Standardization (a minor modification of Weight Standardization (Huang et al., 2017; Qiao et al., 2019)).”
• Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, methodology): “To resolve this, we propose Scaled Weight Standardization, a minor modification of the recently proposed Weight Standardization (Qiao et al., 2019; Huang et al., 2017b), which prevents the growth in the mean signal, leading to a substantial boost in performance.”
• Decorrelated Batch Normalization (2018, methodology): “through re-parameterization techniques [41, 23, 22], Riemannian optimization methods [21, 7], or additional weight regularization [30, 37, 40]—or by designing special scaling coefficients and bias values that can induce normalized activations under certain assumptions [1].”
• Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, background): “, either through re-parameterization [36, 16, 15] or weight regularization [23, 32, 35]), and that specially design either scaling coefficients & bias values [1] or nonlinear function [20], to normalize activation implicitly [39].”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “[51] formally described the re-parameterization idea in training with constraints on weight matrices, and applied it to solve the optimization with zero-mean and unit-norm constraints as shown in Eqn.”