The Effective Number of Parameters: An Analysis of Generalization and Regularization in Nonlinear Learning Systems

By John E. Moody. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “…sets (Craven & Wahba, 1979; Eubank, 1988; Golub, Heath, & Wahba, 1979; Hastie & Tibshirani, 1990; Mosteller & Tukey, 1968; Stone, 1974), Akaike’s information criterion and final prediction error (Akaike, 1970, 1973, 1974), or generalized prediction error (Moody, 1992; Moody & Utans, 1994).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “2 M ’s Training To decrease bitsM + bitsH , we add a regularizing term to E, to punish excessive complexity [4, 5, 91, 294, 155, 135, 97, 170, 169, 104, 290, 7, 290, 87, 286, 319, 100, 102].”
A survey of Bayesian predictive methods for model assessment, selection and comparison (2012, unknown): “The trace term tr [ Ĵ(θ̂)Î−1(θ̂) ] ≤ p was called by Moody (1992) the effective number of parameters.”
Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping (2000, background): “2 Overfitting Much has been written about overfitting and the bias/variance tradeoff in neural nets and other machine learning models [2, 12, 4, 8, 5, 13, 6] .”
Using Recurrent Neural Networks To Forecasting of Forex (2003, background): “One measure of complexity for nonlinear models is P eff , the effective number of parameters (Moody, 1992; Moody, 1994).”
Bankruptcy prediction using support vector machine with optimal choice of kernel function parameters (2005, background): “…depends on researchers’ experience or knowledge to preprocess data in order to select control parameters including relevant input variables, hidden layer size, learning rate, and momentum (Gemen et al., 1992; Lawrence, Giles, & Tsoi, 1997; Moody, 1992; Sarle, 1995; Smith, 1993; Weigend, 1994).”
Connectionist Theory Refinement: Genetically Searching the Space of Network Topologies (1997, methodology): “Two techniques that try to take into account the e ective size of the network areGeneralized Prediction Error (Moody, 1991) and Bayesian methods (MacKay, 1992).”
Expression cloning (2001, unknown): “The number of correspondence inputs is n , ) 2 1 ( − − − = A A tr λ η , and γ is the effective number of parameters [19],”
Neural networks in geophysical applications (2000, methodology): “Hence, several theoretical criteria had to be developed for such nonlinear systems (MacKay, 1992; Moody, 1992; Murata et al., 1994).”
Fast Pruning Using Principal Components (1993, methodology): “The effect of our PCP method is to reduce the rank of each layer of weights in a network by the removal of the least salient eigen-nodes, which reduces the effective number of parameters (Moody, 1992).”
A Bayesian Committee Machine (2000, methodology): “This is analogous to the definition of the effective number of parameters by Wahba (1983), Hastie and Tibshirani (1990), Moody (1992), and MacKay D ow nloaded from http://direct.m it.edu/neco/article-pdf/12/11/2719/814365/089976600300014908.pdf by guest on 02 Septem ber 2021 20 40 60 80 100…”
Overtraining, regularization and searching for a minimum, with application to neural networks (1995, background): “Recently, the concept has also been brought up in connection with neural networks; (for example Poggio and Girosi 1990, Moody 1992, MacKay 1991, Sjoberg and Ljung 1992, Ljung and Sjoberg 1992).”
Quantification of Nitrogen Status in Rice by Least Squares Support Vector Machines and Reflectance Spectroscopy (2009, background): “The overfitting nature is another difficulty to overcome in order to generate a good result (Moody 1992).”
Model Selection in Linear Mixed Models (2013, methodology): “Then the complexity measure in NIC, called the effective number of parameters by Moody (1992), is trace{Kh(θ) · Lh(θ)−1}. Ripley [(1996), page 140] points out that the estimation of this measure is generally not straightforward.”