Batch normalization provably avoids ranks collapse for randomly initialised deep networks

By Hadi Daneshmand, Jonas Köhler, Francis R. Bach, Thomas Hofmann, Aurélien Lucchi. 2020

70 citations recorded by the source corpus.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

1. Ling Shao
2. Jie Qin
3. Yihe Dong
4. Fan Zhu
5. Andreas Loukas
6. Yi Zhou
7. Li Liu
8. Lei Huang

Citations in the corpus

• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth (2021, background): “On the other hand, Daneshmand et al. [DKB + 20] studied rank collapse in randomly initialized linear and ReLU networks and showed that batch normalization is an effective mitigation strategy.”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “Note that the stable-rank [199] of a matrix can be used to implicitly characterize the conditioning of a matrix.”