Dense Associative Memory for Pattern Recognition

By Dmitry Krotov, John J. Hopfield. 2016

520 citations recorded by the source corpus.

A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward neural networks with one hidden layer and various activation functions, which transmit the activities of the visible neurons to the hidden layer. This family of activation functions includes logistics, rectified linear units, and rectified polynomials of higher degrees. The proposed duality makes it possible to apply energy-based intuition from associative memory to analyze computational properties of neural networks with unusual activation functions - the higher rectified polynomials which until now have not been used in deep learning. The utility of the dense memories is illustrated for two test cases: the logical gate XOR and the recognition of handwritten digits from the MNIST data set.

Subfields: Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, background): “Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopﬁeld, 2016).”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “Kortov and Hopfiled [126] examined the existence of adversarial perturbations in the context of Dense AssociativeMemory (DAM)models [162].”
Linear Transformers Are Secretly Fast Weight Programmers (2021, background): “The core component of a Transformer is the self-attention mechanism (Cheng et al., 2016; Parikh et al., 2016; Lin et al., 2017) which was recently connected to the modern Hop-* ﬁeld network (Ramsauer et al., 2021; Krotov & Hopﬁeld, 2016; Demircigil et al., 2017).”
Neural Empirical Bayes (2019, background): “In addition, Hopfield networks have severe limitations in memory capacity, addressed most recently in (Hillar and Tran, 2014; Krotov and Hopfield, 2016; Chaudhuri and Fiete, 2017).”
Adversarial Examples on Object Recognition (2020, background): “ons – each direction of the network has its own biases and the weights are shared. ACM Comput. Surv., Vol. 53, No. 3, Article 66. Publication date: June 2020. 66:26 A. Serban Dense associative models [95] store a set of vectors in memory, corresponding to the learnt patterns. For example, pixel char”
Convolutional Bipartite Attractor Networks (2019, background): “The problem of interpretation is well matched to an early and often overlooked architecture, the attractor net [12, 13, 19, 46].”
Fine-tuning Image Transformers using Learnable Memory (2022, background): “In a recent years, there has been proposed a series of improvements to the energy function of such networks to increase its capacity [24], help protect against adversarial attacks [23] or as a replacement for attention in transformer networks [32].”
Modern Hopfield Networks and Attention for Immune Repertoire Classification (2020, background): “We show that the attention mechanism of transformers is the update rule of modern Hopfield networks (Krotov & Hopfield, 2016, 2018; Demircigil et al., 2017) that are generalized to continuous states in contrast to classical Hopfield networks (Hopfield, 1982).”
Dense Associative Memory Is Robust to Adversarial Inputs (2017, methodology): “he slope of the tanh function. rule Y = tanh h XK =1 F n ˘ i v i+ ˘ 6= ˘ F n ˘ i v i ˘ 6 ˘ i (1) where summation over repeated index iis assumed. The functions F nare rectied polynomials [11] F n(x) = ( xn; x0 0; x<0 (2) As we explained in [11], the argument of the update rule (1) is the diere”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “Nevertheless, Kortov and Hopfiled [297] later provided another evidence based on Dense Associative Memory [298] that supports the role of linearity in neural model susceptibility to adversarial examples.”
Hopfield Networks is All You Need (2020, methodology): “The energy function E in the new type of Hopfield models of Krotov and Hopfield is E = − ∑N i=1 F ( ξxi ) for binary patterns xi and binary state ξ with interaction function F (x) = x, where n = 2 gives the classical Hopfield model [41].”