Statistical mechanics of learning from examples.

By H. Sebastian Seung, Haim Sompolinsky, Naftali Tishby. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

Deep Learning Scaling is Predictable, Empirically (2017, background): “Amari and others show that similar expectations hold for certain models, such as single-and multi-layer perceptrons, and committees of networks (Györgyi and Tishby (1990); Seung et al. (1992); Schwarze and Hertz (1993); Amari and Murata (1993)).”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “The purpose of this work is to tackle continual learning from the complementary perspective of high-dimensional teacher-student models (Gardner & Derrida, 1989; Seung et al., 1992; Biehl & Schwarze, 1993; Zdeborov´a & Krzakala, 2016).”
The committee machine: computational to statistical gaps in learning a two-layers neural network (2018, background): “While the information-theoretic and specialization phase transitions were identi ed in the physics literature on the committee machine [20, 21, 3, 4], the gap between the information-theoretic performance and the performance of AMP —that is conjectured to be optimal among polynomial algorithms— was ”
Multitasking Capability Versus Learning Efficiency in Neural Network Architectures (2017, background): “…of any task-relevant stimulus dimension are mapped onto one of the 5 output units of the task-relevant response dimension corresponds to a non-linear function that was randomly generated 2 with a separate “teacher“ network (cf. Seung, Sompolinsky, & Tishby, 1992), and is the same across tasks.”
Capturing the learning curves of generic features maps for realistic data sets with a teacher-student model (2021, methodology): “Teacher-student models are a popular framework to study the high-dimensional asymptotic performance of machine learning problems with synthetic data, and have been the subject of intense investigations spanning three decades [1, 2, 3, 4, 5, 6, 7].”
Advances in Large Margin Classifiers (2000, methodology): “The first bound on the fat shattering dimension of bounded linear functions in a finite dimensional space was obtained by Shawe-Taylor et al. [1998]. Gurvits [1997] generalized this to infinite dimensional Banach spaces (see Theorem 1.”
Adaptive computation and machine learning (1998, background): “The results and techniques of statistical mechanics are useful in understanding the properties and learning evolution of a number of graphical models used in machine learning [252, 482, 50].”
Efficient noise-tolerant learning from statistical queries (1993, unknown): “There is a voluminous literature on learning perceptrons in general (see the work of Minsky and Papert [1988] for a partial bibliography) and with respect to this distribution in particular [Seung et al. 1992; Baum and Lyuu 1991; Gardner and Derrida 1989].”
Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis. (2012, background): “This question has been studied exhaustively (Gardner 1988, Seung et al. 1992) (see Engel & den Broeck 2001 for an overview), and the general consensus finds that for a wide variety of learning rules, a small generalization error can occur only when the number of examples M is larger than the number…”
Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning (2019, background): “The main interest of such settings is to use the analysis of typical case problems with synthetic data created from simple generative models as means of getting additional insight on real world applications where data are not worst case ([5, 6, 7, 8, 14]).”
Towards Understanding Regularization in Batch Normalization (2018, background): “In general, the expression of free energy under the replica theory is expressed as(Seung et al., 1992)”