Hopfield Networks is All You Need

By Hubert Ramsauer, Bernhard Schafl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Milena Pavlovi'c, Geir Kjetil Ferkingstad Sandve, Victor Greiff and 5 more. 2020

701 citations recorded by the source corpus.

We show that the transformer attention mechanism is the update rule of a modern Hopfield network with continuous states. This new Hopfield network can store exponentially (with the dimension) many patterns, converges with one update, and has exponentially small retrieval errors. The number of stored patterns is traded off against convergence speed and retrieval error. The new Hopfield network has three types of energy minima (fixed points of the update): (1) global fixed point averaging over all patterns, (2) metastable states averaging over a subset of patterns, and (3) fixed points which store a single pattern. Transformer and BERT models operate in their first layers preferably in the global averaging regime, while they operate in higher layers in metastable states. The gradient in transformers is maximal for metastable states, is uniformly distributed for global averaging, and vanishes for a fixed point near a stored pattern. Using the Hopfield network interpretation, we analyzed learning of transformer and BERT models. Learning starts with attention heads that average and then most of them switch to metastable states. However, the majority of heads in the first layers still averages and can be replaced by averaging, e.g. our proposed Gaussian weighting. In contrast, heads in the last layers steadily learn and seem to use metastable states to collect information created in lower layers. These heads seem to be a promising target for improving transformers. Neural networks with Hopfield networks outperform other methods on immune repertoire classification, where the Hopfield net stores several hundreds of thousands of patterns. We provide a new PyTorch layer called "Hopfield", which allows to equip deep learning architectures with modern Hopfield networks as a new powerful concept comprising pooling, memory, and attention. GitHub: this https URL

Subfields: Artificial Intelligence, Cognitive Neuroscience, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Model Architectures

Citations in the corpus

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, methodology): “These modern Hopﬁeld networks for deep learning architectures have an energy function with continuous states and can retrieve samples with only one update (Ramsauer et al., 2021).”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “In the scenario of strict mem- ory requirements, we propose to leverage recently introduced techniques on linear attention (Choromanski et al. 2021b) combined with modern continuous associative memory (MCAM) models (Ramsauer et al. 2021).”
Language Models are Open Knowledge Graphs (2020, methodology): “…(Vaswani et al., 2017) based language models (e.g., BERT and GPT-3), substantial recent work focuses on both visualizing and analyzing the attention (Vig, 2019; Jain & Wallace, 2019; Clark et al., 2019; Michel et al., 2019; Vig et al., 2020; Ramsauer et al., 2020; Hendrycks et al., 2020).”
Online Task-free Continual Learning with Dynamic Sparse Distributed Memory (2022, background): “In its development, SDM respected fundamental biological constraints and is considered to be a generalization of the original Hopfield network [28, 46].”
Large Language Models (2023, background): “13,14, especially to the Hopfield model, may be important [59, 115].”
Modern Hopfield Networks and Attention for Immune Repertoire Classification (2020, methodology): “See Sect.”
On Embeddings for Numerical Features in Tabular Deep Learning (2022, neutral): “Note that we do not include the attention-based models that also apply attention on the level of objects [24, 35, 39], since this non-parametric component is orthogonal to the central topic of our work.”
Transformers learn in-context by gradient descent (2022, background): “Nevertheless, our approach provides an intriguing perspective on, and novel evidence for, an incontext learning mechanism that significantly differs from existing mechanisms based on associative memory (Ramsauer et al., 2020), or by the copying mechanism termed induction heads identified by (Olsson ”
Continual Learning of Large Language Models: A Comprehensive Survey (2024, neutral): “Other memory systems like Hopfield Networks [192] hold promise for future investigation as well.”
Hopular: Modern Hopfield Networks for Tabular Data (2022, methodology): “Furthermore, we test Hopular on UCI datasets (Ramsauer et al., 2021; Klambauer et al., 2017; Wainberg et al., 2016; Fernández-Delgado et al., 2014).”
Long Range Language Modeling via Gated State Spaces (2022, background): “Over the years, a variety of improvements have been proposed to alleviate this quadratic complexity (Tay et al., 2020; Choromanski et al., 2021; Ramsauer et al., 2021; Wang et al., 2020; Katharopoulos et al., 2020; Peng et al., 2021; Ainslie et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020; D”
Data Distributional Properties Drive Emergent In-Context Learning in Transformers (2022, background): “While the hippocampal system is thought to store recent context information in connection weights, whereas transformers store such information directly in their state representations, there is now a body of work pointing out the quantitative and computational equivalence of weight- and state-based r”
Fine-tuning Image Transformers using Learnable Memory (2022, background): “In a recent years, there has been proposed a series of improvements to the energy function of such networks to increase its capacity [24], help protect against adversarial attacks [23] or as a replacement for attention in transformer networks [32].”
Linear Transformers Are Secretly Fast Weight Programmers (2021, background): “Copyright 2021 by the author(s). field network (Ramsauer et al., 2021; Krotov & Hopfield, 2016; Demircigil et al., 2017).”
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation (2021, background): “[52] shows that they are sparser in the later stage of the training.”