Using Fast Weights to Attend to the Recent Past

By Jimmy Ba, Geoffrey E. Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu. 2016

336 citations recorded by the source corpus.

Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower than activities but much faster than the standard weights. These ``fast weights'' can be used to store temporary memories of the recent past and they provide a neurally plausible way of implementing the type of attention to the past that has recently proven helpful in sequence-to-sequence models. By using fast weights we can avoid the need to store copies of neural activity patterns.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, background): “Memory augmentation can also take the form of a recurrent network with Hebbian learning updates (Ba et al., 2016).”
One-Shot Imitation Learning (2017, background): “Fast adaptation has also been achieved through fast-weights (Ba et al., 2016).”
How to train your MAML (2018, background): “Recent work in meta-learning has produced state of the art results in a variety of settings (Wang et al., 2016; Ba et al., 2016; Zoph & Le, 2016; Andrychowicz et al., 2016; Vinyals et al., 2016; Li & Malik, 2016; Li et al., 2017; Antoniou et al 2017; Brock et al., 2017; Munkhdalai & Yu, 2017).”
Meta-Learning with Latent Embedding Optimization (2018, methodology): “The problem of few-shot adaptation has been approached in the context of fast weights (Hinton & Plaut, 1987; Ba et al., 2016), learning-to-learn (Schmidhuber, 1987; Thrun & Pratt, 1998; Hochre-iter et al., 2001; Andrychowicz et al., 2016), and through meta-learning .”
Meta-Learning for Semi-Supervised Few-Shot Classification (2018, methodology): “As future work, we are working on incorporating fast weights (Ba et al., 2016; Finn et al., 2017) into our framework so that examples can have different embedding representations given the contents in the episode.”
Meta Networks (2017, methodology): “Ba et al. (2016) recently used fast weights to replace soft attention mechanism.”
On the Binding Problem in Artificial Neural Networks (2020, background): “Alternatively, certain (Hebbian) forms of fast weights (Schmidhuber, 1992a) can be viewed as a type of internal associative memory based on previous hidden states (Ba et al., 2016).”
How to Represent Part-Whole Hierarchies in a Neural Network (2021, neutral): “The recursive re-use of the neural hardware raises many interesting issues about temporary storage and control ﬂow [Ba et al., 2016] that will only be ﬂeetingly addressed here.”
MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, background): “FWARP [79] focuses on the effect of slow and fast weights changing in learning one sequence-to-sequence network (e.g., long short-term memory network).”
Memory-based Parameter Adaptation (2018, background): “There has been recent work in applying attention to quickly adapt a subset of fast weights (Ba et al., 2016).”
Dynamic Evaluation of Neural Sequence Models (2017, methodology): “Dynamic evaluation as applied at test time, could be considered a form of fast weights (Schmidhuber, 1992; Ba et al., 2016) – recurrent architectures with dynamically changing weight matrices as a function of recent sequence history.”
LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, background): “The concept of generating context-dependent weights is also related to fast weights (Ba et al., 2016).”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “The fast-weights (FW) literature [7, 71] is also related to the idea of selecting different weights at different steps of computation, but with a different motivation for learning context-dependent associative memory [3, 39, 44, 48, 86].”
Multiplicative LSTM for sequence modelling (2016, background): “Dynamic evaluation can be thought of as a type of fast weights memory structure (Ba et al., 2016) that draws the network towards regularities that recently occurred in the sequence, causing the network to assign higher probabilities to these regularities occurring again soon.”
Large-Scale Long-Tailed Recognition in an Open World (2019, methodology): “2) Our memory feature is inspired by meta learning methods with memories [51, 11, 1] to augment the direct feature from the image.”
Random Feature Attention (2021, background): “There are interesting connections between gated R FA and fast weights (Schmidhuber, 1992; 1993; Ba et al., 2016; Miconi et al., 2018, inter alia ).”
Neural Episodic Control (2017, background): “This helps alleviate the typically slow weight updates of stochastic gradient descent applied to the whole network and is reminiscent of work on fast weights (Ba et al., 2016; Hinton & Plaut, 1987), although the architecture we present is quite different.”
Meta-Learning with Warped Gradient Descent (2019, background): “Meta-learning has also been proposed as a separation of parameters into “slow” and “fast” weights, where the latter captures meta-information and the former encapsulates rapid adaptation (Hinton & Plaut, 1987; Schmidhuber, 1992; Ba et al., 2016).”
Differentiable plasticity: training plastic neural networks with backpropagation (2018, methodology): “As in (Wang et al., 2016), we use the Advantage Actor-Critic algorithm(A2C, i.e. a single-threaded, non-parallel variant of the A3C policy gradient search algorithm (Mnih et al., 2016)) to meta-train the network.”