Using fast weights to deblur old memories

By Geoffrey E. Hinton. 1987

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Learning to Learn with Generative Models of Neural Network Checkpoints (2022, methodology): “Rather than parameterizing update rules, neural networks can be used to directly output or modify other neural networks’ parameters (Schmidhuber, 1992; 1993; Hinton & Plaut, 1987).”
On First-Order Meta-Learning Algorithms (2018, background): “It is also related to the older idea of fast weights / slow weights [7].”
Meta-Learning with Latent Embedding Optimization (2018, background): “The problem of few-shot adaptation has been approached in the context of fast weights (Hinton & Plaut, 1987; Ba et al., 2016), learning-to-learn (Schmidhuber, 1987; Thrun & Pratt, 1998; Hochre-iter et al., 2001; Andrychowicz et al., 2016), and through meta-learning .”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “tational framework that effectively generalizes across experiences while retaining speciﬁc memories in a lifelong manner (see Sec. 2.3). An early computational example of this concept was proposed by Hinton and Plaut (1987) in which each synaptic connection has two weights: a plastic weight with slo”
Continual Learning with Deep Generative Replay (2017, background): “Several works illustrate empirical consequences in sequential learning settings [7, 29], and provide a few primitive solutions [16, 30] such as replaying all previous data.”
Meta Networks (2017, background): “Hinton & Plaut (1987) suggested the usage of fast weights for rapid learning.”
Using Fast Weights to Attend to the Recent Past (2016, background): “ically adjusted by the input or hidden activities via the gates. However, LSTMs are still limited to a short-term memory capacity of O(H) for the history of the current sequence. Several researchers [Hinton and Plaut, 1987, Schmidhuber, 1992] have suggested that neural networks could beneﬁt from a t”
Learning Rapid-Temporal Adaptations (2017, background): “The MetaNet udpates synaptic connections (weights) between neurons using fast weights [34, 12] to implement the rapid adaptation.”
Measuring Catastrophic Forgetting in Neural Networks (2017, background): “Since then, there have been multiple attempts to mitigate this phenomenon (Hinton and Plaut 1987; Robins 1995; Goodrich and Arel 2014; Draelos et al. 2016; Ren et al. 2017; Fernando et al. 2017; Kirkpatrick et al. 2017).”
Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks (2017, background): “, 1982) and various constraints (Miller and Mackay, 1994), or more advanced update dynamics such as dual weights for fast and slow decay (Levy and Bairaktaris, 1995; Hinton and Plaut, 1987; Bullinaria, 2009a; Soltoggio, 2015).”
Short-term plasticity as cause–effect hypothesis testing in distal reward learning (2014, background): “Such as distinction is not new in connectionist models (Hinton and Plaut, 1987; Schmidhuber, 1992; Levy and Bairaktaris, 1995; Tieleman and Hinton, 2009; Bullinaria, 2009), however, in the proposed study the idea is extended to model hypothesis testing and memory consolidation with distal rewards.”
Memory Efficient Experience Replay for Streaming Learning (2018, background): “Regularization constrains weight updates so they do not interfere with previous learning [35], [42], [48],”
Few-Shot and Continual Learning with Attentive Independent Mechanisms (2021, methodology): “Earlier works are based on regularization method, with [17] proposing the use of fast and slow training weights, borrowing the idea of plasticity and stability for network training.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “Another alternative to CLS theory is the idea of using both fast and slow weights between units for each connection in the network (Hinton & Plaut, 1987).”
MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, background): “As for the slow and fast weights learning, FWDOM [78] assumes that each connection of a network has two weights (a plastic weight and elastic weight), and focuses on the effect of slow and fast changing of these weights in learning one network.”
Lookahead Optimizer: k steps forward, 1 step back (2019, methodology): “Lookahead first updates the “fast weights” [12] k times using any standard optimizer in its inner loop before updating the “slow weights” once in the direction of the final fast weights.”