Three unfinished works on the optimal storage capacity of networks

By E. Gardner, Bernard Derrida. 1989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Online Bag-of-Visual-Words Generation for Unsupervised Representation Learning (2020, background): “This paradigm has a long research history [25, 62] and it is frequently used for distilling a single large network or an ensemble, the teacher, into a smaller network, the student [5, 39, 44, 54].”
Statistical physics of inference: thresholds and algorithms (2015, background): “In physics a case of perceptron that was studied in detail is that of binary synaptic weights xi ∈ {±1} [83, 84, 94].”
Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning (2019, background): “Spherical perceptron The most commonly studied model ([9, 36, 19, 9]) with continuous weights is the spherical model with w ∈ Rd such that ‖w‖(2)2 = d.”
Local entropy as a measure for sampling solutions in constraint satisfaction problems (2015, methodology): “This problem has been extensively analyzed in the limit of large N by means of Replica [10, 11] and Cavity [12] methods, finding that in the typical case there is an exponential (in N) number of solutions up to a critical capacity αc = 0.”
Capturing the learning curves of generic features maps for realistic data sets with a teacher-student model (2021, methodology): “This special case has been extensively studied in the statistical physics community using the heuristic replica method [23, 24, 1, 2, 3].”
OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning (2020, background): “This paradigm has a long research history [24, 58] and it is frequently used for distilling a single large network or an ensemble, the teacher, into a smaller network, the student [5, 36, 41, 51].”
Rigorous Learning Curve Bounds from Statistical Mechanics (2005, methodology): “This bound on the critical value was known from the work of Gardner and Derrida [12], and extended to the case of boolean inputs by Baum, Lyuu and Rivin [2, 19].”
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup (2019, unknown): “On the other hand, there has been a considerable body of theoretical work calculating the generalisation ability of neural networks for data arising from a probabilistic model, particularly within the framework of statistical mechanics 17,18,20–22 .”
Modelling the influence of data structure on learning in neural networks (2019, background): “A classic model for data sets is the vanilla teacher-student setup [11], where unstructured i.”
Efficient noise-tolerant learning from statistical queries (1993, unknown): “There is a voluminous literature on learning perceptrons in general (see the work of Minsky and Papert [1988] for a partial bibliography) and with respect to this distribution in particular [Seung et al. 1992; Baum and Lyuu 1991; Gardner and Derrida 1989].”
Machine learning and the physical sciences (2019, methodology): “This line of work was pioneered by Elisabeth Gardner (Gardner and Derrida, 1989) and actively studied in physics in the past for special cases of Pout and PW , see e.g. (Györgyi and Tishby, 1990; Seung et al., 1992a; Sompolinsky et al., 1990).”
Optimal errors and phase transitions in high-dimensional generalized linear models (2017, methodology): “This perceptron model has already been studied in physics [22] and more recently in statistics, where it is known as “one-bit compressed sensing” [6, 15].”