A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks

By K. Müller, Michael Finke, Noboru Murata, Klaus Schulten, Shun-ichi Amari. 1996

The universal asymptotic scaling laws proposed by Amari et al. are studied in large scale simulations using a CM5. Small stochastic multilayer feedforward networks trained with backpropagation are investigated. In the range of a large number of training patterns t, the asymptotic generalization error scales as 1/t as predicted. For a medium range t a faster 1/t2 scaling is observed. This effect is explained by using higher order corrections of the likelihood expansion. It is shown for small t that the scaling law changes drastically, when the network undergoes a transition from strong overfitting to effective learning.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Stefan L. Frank
2. Peter Alexander Jansen

Citations in the corpus

• Asymptotic statistical theory of overtraining and cross-validation (1997, background): “In an intermediate phase (cf. [24]) of larger than simulations show that cross-validation stopping is effective in general.”
• Learn more by training less: systematicity in sentence processing by recurrent networks (2006, background): “…as overfitting occurs: the network has so many free parameters (i.e. trainable connection weights) relative to the size of the training set that all training examples can simply be individually stored, resulting in impoverished generalization (see, e.g. Baum and Haussler 1989, Müller et al. 1996).”