Statistical Mechanics of Learning

By A. Engel, C. Broeck. 2001

631 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “In the perceptron, the simplest network without hidden layers, the cusp in the test error at the jamming point is also observed and predicted analytically [50, 51, 52, 53, 54, 55].”
High-dimensional dynamics of generalization error in neural networks (2017, methodology): “This is related to a large body of prior work on shallow networks (Chauvin, 1990; P. Baldi and Chauvin, 1991; LeCun, Kanter, and Solla, 1991; Seung, Sompolinsky, and Tishby, 1992; Dodier, 1995; P.F. Baldi and Hornik, 1995; Kinouchi and Caticha, 1995; Dunmur and Wallace, 1999; Krogh and Hertz, 1992;…”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “…has a long history in studying the dynamics of learning in neural network models (Gardner & Derrida, 1989; Seung et al., 1992; Watkin et al., 1993; Engel & Van den Broeck, 2001; Zdeborov ´ a & Krzakala, 2016) and has recently experienced a surge of activity in the machine learning community…”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “[39] Andreas Engel and Christian Van den Broeck.”
Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses. (2015, background): “245 , after which there is a first-order transition and only one solution is possible: the teacher itself [1,10].”
Statistical Mechanics of On-line Learning (2009, background): “We can only refer to the list of references, in particular [27] and [28] may serve as a starting point for the interested reader.”
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, background): “In the past, statistical mechanics provided a useful theoretical framework for such typical-case analyses for various algorithms [22, 23, 24, 25, 26, 27, 28, 29].”
Jamming in Multilayer Supervised Learning Models. (2018, background): “For example, it is known for Parity machine that the phase diagram undergoes first a RS-to-1RSB transition [7].”
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks (2020, methodology): “This is a non-rigorous, but a standard technique in the physics of disordered systems (M ´ ezard et al., 1987; Engel & Van den Broeck, 2001).”
Capturing the learning curves of generic features maps for realistic data sets with a teacher-student model (2021, methodology): “Teacher-student models are a popular framework to study the high-dimensional asymptotic performance of machine learning problems with synthetic data, and have been the subject of intense investigations spanning three decades [1, 2, 3, 4, 5, 6, 7].”
Optimal errors and phase transitions in high-dimensional generalized linear models (2017, neutral): “23–25 for reviews.”
Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks (2019, background): “HT-MU applies to the analysis of complicated systems, including many physical systems, traditional NNs [23, 24], and even models of the dynamics of actual spiking neurons.”
Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning (2019, methodology): “1, and we also refer the reader to the relevant literature in physics ([18, 26, 8, 27, 14]) and in mathematics ([17, 23, 28, 29, 30]).”
Triple descent and the two kinds of overfitting: where and why do they appear? (2020, background): “h double descent has recently gained much interest in the context of deep learning, a seemingly similar phenomenon has been well-known for several decades for simpler models such as linear regression [13, 14, 20, 21]. However, in the context of linear models, the number of parameters Pis not a free ”
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup (2019, neutral): “[18] A.”