Linear Transformers Are Secretly Fast Weight Programmers

By Imanol Schlag, Kazuki Irie, J. Urgen Schmidhuber. International Conference on Machine Learning, 2021

474 citations recorded by the source corpus.

We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow"neural net learns by gradient descent to program the ``fast weights"of another net through sequences of elementary programming instructions which are additive outer products of self-invented activation patterns (today called keys and values). Such Fast Weight Programmers (FWPs) learn to manipulate the contents of a finite memory and dynamically interact with it. We infer a memory capacity limitation of recent linearised softmax attention variants, and replace the purely additive outer products by a delta rule-like programming instruction, such that the FWP can more easily learn to correct the current mapping from keys to values. The FWP also learns to compute dynamically changing learning rates. We also propose a new kernel function to linearise attention which balances simplicity and effectiveness. We conduct experiments on synthetic retrieval problems as well as standard machine translation and language modelling tasks which demonstrate the benefits of our methods.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/1a703f08da01cf737cce3fb9064259b3f4b44e9c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Transformers learn in-context by gradient descent (2022, methodology): “…to the model learning to implement a copying layer, see section 4 and proposition 3, and allows us to provide a simple and different construction to Schlag et al. (2021) that solely is built on the standard linear, and approximately softmax, self-attention layer but still implements gradient…”
Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024, unknown): “The most relevant work is DeltaNet [62], which is equivalent to TTT-Linear with inner-loop mini-batch size 1, without the Layer Norm and residual connection.”
Transformers learn to implement preconditioned gradient descent for in-context learning (2023, methodology): “Following (Schlag et al., 2021; von Oswald et al., 2023), we consider the linear self-attention layer.”
On the link between conscious function and general intelligence in humans and machines (2022, background): “Transformers can also be linked to earlier work on “fast weights” (Schlag et al., 2021), where a portion of the “synaptic weights” within the network adapt online to current inputs, without requiring further training (Hinton & Plaut, 1987; Schmidhuber, 1992).”
Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, unknown): “On the other hand, the delta rule has been shown to have better memory capacity [27, 83, 54, 97].”
Choose a Transformer: Fourier or Galerkin (2021, methodology): “Inspired by many insightful pioneering work in Transformers [50, 19, 75, 84, 96, 97, 95, 59, 76, 66], we have modiﬁed the attention mechanism minimally yet in a mathematically profound manner to better serve the purpose of operator learning.”
Gated Linear Attention Transformers with Hardware-Efficient Training (2023, methodology): “For better memory utilization, Schlag et al. (2021) use the delta rule to edit the memory dynamically.”
What learning algorithm is in-context learning? Investigations with linear models (2022, background): “Different theoretical constructions, in the context of meta-learning, have been shown for linear self-attention models (Schlag et al., 2021), or for other neural architectures such as recurrent neural networks (Kirsch & Schmidhuber, 2021).”
Hybrid Random Features (2021, background): “Schlag et al. (2021) replaced random features by sparse deterministic constructions (no longer approximating softmax kernel).”
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers (2023, background): “Numerous linear attention mechanisms have also been proposed (Katharopoulos et al., 2020; Schlag et al., 2021; Choromanski et al., 2020), although we are not aware of competitive results on large scale language modeling tasks.”
Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, unknown): “The delta update rule (Widrow et al., 1960; Schlag et al., 2021b) dynamically erases the value ( v old t ) associated with the current input key ( k t ) and writes a new value ( v new t ), which is a linear combination of the current input value and the old value based on the “writing strength” β t…”
Quasi Linear Kernel Attention with Infinite Capacity (2026, unknown): “Its definition is inspired by the associative recall experiment of Schlag et al. (2021) and measures up to which sequence length the identity can be approximated by the attention algorithm.”
EmbeddingRWKV: State-Centric Retrieval with Reusable States (2026, unknown): “…the transition matrix W t is further refined through dynamic evolution, typically expressed by diagonal-izing a vector w t and incorporating a low-rank up-date: where ˆ κ t and a t are fast weights (Schlag et al., 2021) that enhance the model’s ability to selectively update its internal memory.”