THE STATISTICAL-MECHANICS OF LEARNING A RULE

By T. Watkin, A. Rau, Michael Biehl. 1993

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

High-dimensional dynamics of generalization error in neural networks (2017, methodology): “tudent, given the complexity of the problem and the number of raining samples available? We begin by simulating the dynamics in a student–teacher cenario (Engel & Van Den Broeck, 2001; Saad & Solla, 1995; eung et al., 1992; Watkin et al., 1993; Wei & Amari, 2008; Wei, hang, Cousseau, Ozeki, & Amari,”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “…framework has a long history in studying the dynamics of learning in neural network models (Gardner & Derrida, 1989; Seung et al., 1992; Watkin et al., 1993; Engel & Van den Broeck, 2001; Zdeborov ´ a & Krzakala, 2016) and has recently experienced a surge of activity in the machine…”
The committee machine: computational to statistical gaps in learning a two-layers neural network (2018, background): “While the information-theoretic and specialization phase transitions were identified in the physics literature on the committee machine [3, 4, 20, 21], the gap between the information-theoretic performance and the performance of AMP—that is conjectured to be optimal among polynomial algorithms—was n”
Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning (2019, background): “The main interest of such settings is to use the analysis of typical case problems with synthetic data created from simple generative models as means of getting additional insight on real world applications where data are not worst case ([5, 6, 7, 8, 14]).”
Incremental on-line learning: A review and comparison of state of the art algorithms (2018, background): “Massive theoretical work has been done to evaluate their generalization ability and convergence speed in the stationary setting [9, 10], often accompanied by assumptions such as linearly separable data [11].”
Inverse statistical problems: from the inverse Ising problem to data science (2017, background): “Such a perspective is not new to statistical physics; the analysis of neural networks in the seventies and eighties of the last century led to a statistical mechanics of learning [72, 95, 238], where the phase space is defined by the set of possible rules linking the input into a machine with an out”
A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors (2002, methodology): “, [32], [10], [16] Since for integer is nothing but the normalizing coefficient for a system consisting ofreplicas of the posterior probability (6) sharing the sameand , we can write down explicitly as shown in (20) at the bottom of the page, where we have introduced replicated random variables”
Modelling the influence of data structure on learning in neural networks (2019, background): “…played a major role in theoretical studies of the generalisation ability of neural networks from an average-case perspective, particularly within the framework of statistical mechanics [3, 12, 4, 5, 13, 14, 15, 16, 17], and also in recent statistical learning theory works, e.g. [18, 19, 20, 21].”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, methodology): “From the SMTOG perspective, NN overfitting,40 which results in NN overtraining,41 is an example of non-self-averaging.”
Incremental learning algorithms and applications (2016, background): “Interestingly, online learning has already very early been accompanied by their exact mathematical investigations [162].”
Tree Induction vs. Logistic Regression: A Learning-Curve Analysis (2003, unknown): “They have been analyzed theoretically, using statistical mechanics (Watkin, Rau, and Biehl, 1993; Haussler, Kearns, Seung, and Tishby, 1996). of the algorithms for classification (zero/one loss).”