The space of interactions in neural network models

By E. Gardner. 1988

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “[14] Elizabeth Gardner, “The space of interactions in neural network models,” Journal of physics A: Mathematical and general 21, 257 (1988).”
Using Fast Weights to Attend to the Recent Past (2016, methodology): “…of the weights by using an iterative, error correction storage rule to learn weights that can retrieve each bit of a pattern from all the other bits [Gardner, 1988], but for our purposes maximizing the capacity is less important than having a simple, non-iterative storage rule, so we will use an…”
Capacity and Trainability in Recurrent Neural Networks (2016, methodology): “A foundational result in machine learning is that a single-layer perceptron with N 2 parameters can store at least 2 bits of information per parameter (Cover, 1965; Gardner, 1988; Baldi & Venkatesh, 1987).”
Geometric compression of invariant manifolds in neural nets (2020, methodology): “In the physics literature, β has been computed in regression or classification tasks for fixed kernels [25, 16, 26, 27, 28].”
Hebbian learning, its correlation catastrophe, and unlearning (1997, background): “provided pD 0:5 (or aD 0). As N becomes large, the numberqmax of patterns that can be stored and retrieved scales with N but one has to face the fact [17] that qmax N . That is to say, the storage capacity of the Hopfield model is Hopfield c D qmax=ND 0:14, which is to be contrasted with the theoret”
Catastrophic Forgetting and the Pseudorehearsal Solution in Hopfield-type Networks (1998, background): “The theoretical maximum amount of information that can be stored in the Hopfield network, independent of the learning rule, is M cap = N patterns for networks with symmetric connections, and 2N for asymmetric connections (Gardner, 1988).”
Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, unknown): “On the other hand, the delta rule has been shown to have better memory capacity [27, 83, 54, 97].”
Machine learning and the physical sciences (2019, methodology): “A much tighter application to learning models was made by the seminal work of Elizabeth Gardner who applied the replica trick (Gardner, 1987, 1988) to calculate volumes in the weights space for simple feed-forward neural networks, for both supervised and unsupervised learning 7 models.”
On the geometry of generalization and memorization in deep neural networks (2021, background): “MFTMA framework is a generalization of statistical mechanical theory of perceptron capacity for discrete points (Gardner, 1988) to manifolds.”
Optimal degrees of synaptic connectivity (2017, unknown): “We considered the simple case of learning associations between outputs and random, uncorrelated input patterns with a fixed coding level, a standard benchmark for learning systems (Gardner, 1988; Barak et al., 2013; Babadi and Sompolinsky, 2014).”
Statistical Mechanics of Optimal Convex Inference in High Dimensions (2016, background): “the number of unknowns, for example, in neural network memory capacity [12], perceptron learning theory [13,14], communication theory [15], compressed sensing [16–19],”
Statistical mechanics of deep learning (2024, unknown): “…of r → 0, the problem is reduced to linear separation of P random points, the centers; hence, the capacity is given by Gardner’s formula, which is the maximum load for separation of random i.i.d. points with a margin κ given by the Gardner theory [10], with Gaussian measure Dt = 1 √ 2 π e − t 2 2 .”
Statistical mechanics of complex neural systems and high dimensional data (2013, background): “A key conceptual advance made by Gardner [79, 80] was that this same machinery could be applied to the analysis of learning, by performing statistical mechanics directly on the space of synaptic connectivities, with the training examples presented to the system playing the role of quenched disorder.”
Machine Learning for Fluid Mechanics (2019, background): “Another source of stimulus were the works by Hopﬁeld (1982); Gardner (1988); Hinton & Sejnowski (1986) who developed links between machine learning algorithms and statistical mechanics.”
Neural Associative Memory with Optimal Bayesian Learning (2011, background): “…covariance rule, the optimal Bayesian model reaches the Gardner bound for sparse memory patterns p, q → 0 in the limit Mpq → ∞ corresponding to moderately sparse patterns with mp log(n) where the network can store C = 1/(2 ln 2) ≈ 0.72 bps (compare equation 37 to equation 40 in Gardner, 1988).”