Neural networks and physical systems with emergent collective computational abilities.

By John J. Hopfield. 1982

Computational properties of use of biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons). The physical meaning of content-addressable memory is described by an appropriate phase space flow of the state of a system. A model of such a system is given, based on aspects of neurobiology but readily adapted to integrated circuits. The collective properties of this model produce a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size. The algorithm for the time evolution of the state of the system is based on asynchronous parallel processing. Additional emergent collective properties include some capacity for generalization, familiarity recognition, categorization, error correction, and time sequence retention. The collective properties are only weakly sensitive to details of the modeling or the failure of individual devices.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Gaussian Error Linear Units (GELUs) (2016, methodology): “Early artificial neurons utilized binary threshold units (Hopfield, 1982; McCulloch & Pitts, 1943).”
Residual Energy-Based Models for Text Generation (2020, background): “Energy-based models have a long history in machine learning (Hopfield, 1982; Hinton, 2002; LeCun et al., 2006; Ranzato et al., 2007).”
Neural Turing Machines (2014, background): “This is related to the content-addressing of Hopﬁeld networks (Hopﬁeld, 1982).”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, background): “This mode seeking and mode shifting interpretation is related to Hopfield network [60] that uses local modes of energy landscape to memorize the observed examples.”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, background): “This includes the attractor dynamics popular in computational neuroscience [20, 2, 36].”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, methodology): “The Hopfield network [16] adapted these physical energy models into a model capable of representing arbitrary observed data.”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, background): “It can also be considered a form of associative memory [13].”
A Theory of Generative ConvNet (2016, background): “Proposition 2 is related to the Hopfield network (Hopfield, 1982).”
CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, background): “Hopﬁeld networks are energy-based, binary associative memories, which popularized artiﬁcial neural networks in the 1980s (Amari, 1972; Hopﬁeld, 1982, 1984).”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Unsupervised learning methods span a very broad spectrum of approaches going back to the roots of artiﬁcial neural networks [53, 39, 2, 58, 27, 41, 28, 30, 25], and are too numerous to fully review here.”
The Graph Neural Network Model (2009, neutral): “[47]–[50] and Hopfield neural networks [51].”
Deep learning in neural networks: An overview (2014, background): “…simple NNs trained by SL (e.g., Narendra & Thathatchar, 1974; Rosenblatt, 1958, 1962; Widrow & Hoff, 1962) and UL (e.g., Grossberg, 1969; Kohonen, 1972; von der Malsburg, 1973; Willshaw & von der Malsburg, 1976), as well as closely related associative memories (e.g., Hopfield, 1982; Palm, 1980).”
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, background): “Techniques like support vector machines [34], and more recently deep learning [35]—revisiting neural networks architectures [36] are common choices to learn supervised models from data.”
Using Fast Weights to Attend to the Recent Past (2016, background): “A potential problem with fast associative memory is that the scalar product of two hidden vectors could vanish or explode depending on the norm of the hidden vectors.”
On the Binding Problem in Artificial Neural Networks (2020, background): “Early work includes Hopfield networks (Hopfield, 1982), Boltzmann machines (Ackley et al., 1985), and associative memory (Kohonen, 1989).”
Statistical inference using weak chaos and infinite memory (2010, methodology): “Now let’s generalize the one neuron model to a network of N neurons (a Hopfield network (Hopfield, 1982)).”
Structural Synaptic Plasticity Has High Memory Capacity and Can Explain Graded Amnesia, Catastrophic Forgetting, and the Spacing Effect (2014, background): “First, it is well understood that weight capacity of biologically plausible memory models is limited by hard theoretical bounds suggesting Cwpv0:72 bit per synapse even for an infinite computing precision with N??”
Generalization Through the Recurrent Interaction of Episodic Memories (2012, background): “…and others, views the hippocampus to be a content-addressable autoassociative memory system supporting the rapid storage and retrieval of individual episodes (Burgess, 2006; Hasselmo, 1999; Hopfield, 1982; Marr, 1971; McNaughton & Morris, 1987; O’Reilly & McClelland, 1994; Treves & Rolls, 1992).”
Supermasks in Superposition (2020, methodology): “We introduce an extension to the basic SupSup algorithm that stores supermasks implicitly as attractors in a fixed-size Hopfield network [20] (Section 3.”