Tensor Programs III: Neural Matrix Laws

By Greg Yang. 2020

64 citations recorded by the source corpus.

In a neural network (NN), \emph{weight matrices} linearly transform inputs into \emph{preactivations} that are then transformed nonlinearly into \emph{activations}. A typical NN interleaves multitudes of such linear and nonlinear transforms to express complex functions. Thus, the (pre-)activations depend on the weights in an intricate manner. We show that, surprisingly, (pre-)activations of a randomly initialized NN become \emph{independent} from the weights as the NN's widths tend to infinity, in the sense of \emph{asymptotic freeness} in random matrix theory. We call this the \emph{Free Independence Principle (FIP)}, which has these consequences: 1) It rigorously justifies the calculation of asymptotic Jacobian singular value distribution of an NN in Pennington et al. [36,37], essential for training ultra-deep NNs [48]. 2) It gives a new justification of \emph{gradient independence assumption} used for calculating the \emph{Neural Tangent Kernel} of a neural network. FIP and these results hold for any neural architecture. We show FIP by proving a Master Theorem for any Tensor Program, as introduced in Yang [50,51], generalizing the Master Theorems proved in those works. As warmup demonstrations of this new Master Theorem, we give new proofs of the semicircle and Marchenko-Pastur laws, which benchmarks our framework against these fundamental mathematical results.

Subfields: Artificial Intelligence, Computational Mathematics, Statistics and Probability, Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Weizhu Chen
3. Xiaodong Liu
4. J. Edward Hu
5. Greg Yang
6. Mohit Bansal
7. Arnaud Doucet
8. Soufiane Hayou

In Computational Mathematics

1. J. Edward Hu
2. Judith Rousseau
3. Arnaud Doucet
4. Soufiane Hayou

In Statistics and Probability

1. J. Edward Hu
2. Judith Rousseau
3. Arnaud Doucet
4. Soufiane Hayou
5. Jianfeng Gao
6. Greg Yang
7. I. Castillo
8. Nicolas Chopin

In Scaling & Learning Theory

1. J. Edward Hu
2. Soufiane Hayou
3. Arnaud Doucet
4. Judith Rousseau
5. Jianfeng Gao
6. Greg Yang
7. Sam Schoenholz
8. Jeffrey Pennington

In Statistical Learning Theory

1. J. Edward Hu
2. Arnaud Doucet
3. Judith Rousseau
4. Soufiane Hayou
5. Weizhu Chen
6. Jianfeng Gao
7. Xiaodong Liu
8. David Farhi

Citations in the corpus

• Feature Learning in Infinite-Width Neural Networks (2020, methodology): “See (Yang, 2020b).”
• Tensor Programs II: Neural Tangent Kernel for Any Architecture (2020, unknown): “[64] Greg Yang.”
• Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (2022, background): “The intuitions we shall describe regarding SGD can be made rigorous as in [53, 54]; those regarding Adam are new, and their formalization will be done in an upcoming paper.”
• Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks (2019, methodology): “• For ResNets, we also have convergence to a trivial NTK regime but this always occurs at a polynomial rate, irrespective of the initialization.”