Statistical Theory of Learning Curves under Entropic Loss Criterion

By Shun-ichi Amari, Noboru Murata. 1993

The present paper elucidates a universal property of learning curves, which shows how the generalization error, training error, and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases. The error is measured by the entropic loss. It is proved that the generalization error converges to H0, the entropy of the conditional distribution of the true machine, as H0 + m/(2t), while the training error converges as H0 - m/(2t), where t is the number of examples and m shows the complexity of the network. When the model is faithful, implying that the true machine is in the model, m is reduced to m, the number of modifiable parameters. This is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator. Similar relations are obtained for the Bayes and Gibbs learning algorithms. These learning curves show the relation among the accuracy of learning, the complexity of a model, and the number of training examples.

Subfields: Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

Deep Learning Scaling is Predictable, Empirically (2017, background): “Amari and others show that similar expectations hold for certain models, such as single-and multi-layer perceptrons, and committees of networks (Györgyi and Tishby (1990); Seung et al. (1992); Schwarze and Hertz (1993); Amari and Murata (1993)).”
Deep learning in neural networks: An overview (2014, background): “, 1979), final prediction error (Akaike, 1970), generalized prediction error (Moody and Utans, 1994; Moody, 1992); see also (Holden, 1994; Wang et al., 1994; Amari and Murata, 1993; Wang et al., 1994; Guyon et al., 1992; Vapnik, 1992; Wolpert, 1994).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “2 M ’s Training To decrease bitsM + bitsH , we add a regularizing term to E, to punish excessive complexity [4, 5, 91, 294, 155, 135, 97, 170, 169, 104, 290, 7, 290, 87, 286, 319, 100, 102].”
MUTUAL INFORMATION, METRIC ENTROPY AND CUMULATIVE RELATIVE ENTROPY RISK (1997, background): “Amari has developed an extensive theory that relates the risk when is the true state of Nature to certain di erential-geometric properties of the parameter space in the neighborhood of involving Fisher information and related quantities [2, 3] (see also [57, 39, 55]).”
Adaptive computation and machine learning (1998, methodology): “It is also called the Fisher information matrix (see also [5, 16, 373]).”
A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks (1996, methodology): “1) holds for general stochastic machines with smooth outputs (Amari and Murata 1993; Murata et al. 1993).”
Natural Gradient Works Eciently in Learning (result): “The results can be stated in terms of the generalization error instead of the covariance of the estimator, and we can obtain more universal results (see Amari, 1993; Amari and Murata, 1993).”
Natural Gradient Works Efficiently in Learning (1998, result): “The results can be stated in terms of the generalization error instead of the covariance of the estimator, and we can obtain more universal results (see Amari, 1993; Amari & Murata, 1993).”
Fast and Accurate Uncertainty Estimation in Chemical Machine Learning. (2018, unknown): “6,7 Then, the trained neural network competes with the accuracy of the underlying electronic structure method despite having a computational expense that might be many orders of magnitude smaller.”