On a Model of Associative Memory with Huge Storage Capacity

By Mete Demircigil, J. Heusel, Matthias Löwe, Sven Upgang, F. Vermet. 2017

233 citations recorded by the source corpus.

In Krotov et al. (in: Lee (eds) Advances in Neural Information Processing Systems, Curran Associates, Inc., Red Hook, 2016) Krotov and Hopfield suggest a generalized version of the well-known Hopfield model of associative memory. In their version they consider a polynomial interaction function and claim that this increases the storage capacity of the model. We prove this claim and take the ”limit” as the degree of the polynomial becomes infinite, i.e. an exponential interaction function. With this interaction we prove that model has an exponential storage capacity in the number of neurons, yet the basins of attraction are almost as large as in the standard Hopfield model.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Model Architectures

Citations in the corpus

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, background): “Modern Hopfield networks have much higher storage capacity (Demircigil et al., 2017; Krotov & Hopfield, 2016) than classical Hopfield networks (Hopfield, 1982; 1984).”
Modern Hopfield Networks and Attention for Immune Repertoire Classification (2020, background): “We show that the attention mechanism of transformers is the update rule of modern Hopfield networks (Krotov & Hopfield, 2016, 2018; Demircigil et al., 2017) that are generalized to continuous states in contrast to classical Hopfield networks (Hopfield, 1982).”
Hopfield Networks is All You Need (2020, methodology): “It has been shown that the network converges even after one update [23].”
Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, unknown): “Later works in Hopﬁeld networks use higher-order polynomials [21] and exponential kernels [92, 49] to enhance the memory capacity, which is also related to attention with polynomial kernels explored in PolysketchFormer [44] and Based Linear Attention [6, 1].”
Linear Transformers Are Secretly Fast Weight Programmers (2021, background): “The core component of a Transformer is the self-attention mechanism (Cheng et al., 2016; Parikh et al., 2016; Lin et al., 2017) which was recently connected to the modern Hop-* ﬁeld network (Ramsauer et al., 2021; Krotov & Hopﬁeld, 2016; Demircigil et al., 2017).”