Eigenvalues of covariance matrices: Application to neural-network learning.

By Y. L. Cun, I. Kanter, Sara A. Solla. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “In the perceptron, the simplest network without hidden layers, the cusp in the test error at the jamming point is also observed and predicted analytically [50, 51, 52, 53, 54, 55].”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “them (Desjardins et al., 2015). An alternative to centering is to push the mean activation toward zero by an appropriate activation function. Therefore tanhhas been preferred over logistic functions (LeCun et al., 1991; 1998). Recently “Leaky ReLUs” (LReLUs) that replace the negative part of the ReL”
Measuring and regularizing networks in function space (2018, background): “ize and whiten activations or gradients have been forwarded in the literature (Raiko et al. (2012);Simard et al. (1998); Schraudolph & Sejnowski (1996); Crammer et al. (2009); Wang et al. (2013); LeCun et al. (1991); Schraudolph (1998); Salimans & Kingma (2016)). A similar effect is able to ”
Random Matrix Theory and Wireless Communications (2004, background): “as late as 1991, rediscoveries of the Marc̆enko-Pastur law can be found in the Physics literature [50].”
Object Recognition with Gradient-Based Learning (1999, background): “This makes the mean input roughly 0, and the variance roughly 1 which accelerates learning [ 20 ].”
Neural networks in geophysical applications (2000, background): “For instance, Le Cun et al. (1991) show that correcting each individual input value for the mean and standard deviation of this component in the training set will increase the learning speed.”
Neural Networks: Tricks of the Trade (2012, background): “(finding good solutions for task I3 appears to extremely difficult, as Roosen and Hastie [12] also discovered.”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “(2)As a complementary point, a cusp in generalization has been found in several perceptron problems when the ratio of number of training examples by the number of parameters is tuned [37, 12, 6], see also [27].”
Gradient-based learning applied to document recognition (1998, background): “This makes the mean input roughly zero and the variance roughly one, which accelerates learning [46].”
Centering Neural Network Gradient Factors (1996, methodology): “Centered. As suggested by LeCun et al. [ 4 ], the activity of the network’s input and hidden units should be centered to permit faster learning.”
Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, background): “MacKay (1992b) and MacKay (2003) contains a further discussion of the connection between Occam factors and minimum description length.”