Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks

By J. Urgen Schmidhuber. 1992

Subfields: Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Model Architectures

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “, 2016), augment models with special memory mechanisms (Santoro et al., 2016; Schmidhuber, 1992), and maximize the degree to which models can learn new tasks (Finn et al.”
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, background): “A popular approach for meta-learning is to train a meta-learner that learns how to up-date the parameters of the learner’s model (Bengio et al., 1992; Schmidhuber, 1992; Bengio et al., 1990).”
HyperNetworks (2016, background): “Even before the work on HyperNEAT and DCT, Schmidhuber (1992; 1993) has suggested the concept of fast weights in which one network can produce context-dependent weight changes for a second network.”
One-Shot Imitation Learning (2017, background): “Meta-learning has also been studied to discover neural network weight optimization algorithms [8, 9, 23, 50, 2, 31].”
Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (2019, methodology): “This embedding is a Tensor-Product Representation ( TPR ; Smolensky, 1990) in an end-to-end-differentiable TPR system (Schlag & Schmidhuber, 2018; Schmidhuber, 1992) that learns “internal spotlights of attention” (Schmidhu-ber, 1993).”
Learning multiple visual domains with residual adapters (2017, background): “A few authors [34, 16, 2], in particular, have proposed to learn neural networks that predict, in a data-dependent manner, the parameters of another.”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “Few-shot learning [25, 35, 46, 53] has recently been primarily addressed in the meta-learning framework [57, 71, 80], where a model learns an update rule for the parameters of a base-learner model [6, 7, 72] through a sequence of training episodes [79, 88].”
Meta-learning with differentiable closed-form solvers (2018, methodology): “This technique has been first introduced by Schmidhuber [35, 36] and recently revamped by Bertinetto et al.”
Learning feed-forward one-shot learners (2016, background): “The idea of parameter prediction was, to our knowledge, first explored by Schmidhuber [20] in a recurrent architecture with one network that modifies the weights of another.”
Dynamic Neural Networks: A Survey (2021, neutral): “This idea was first suggested in [116], where both the weight prediction model and the backbone”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, methodology): “Rather than parameterizing update rules, neural networks can be used to directly output or modify other neural networks’ parameters (Schmidhuber, 1992; 1993; Hinton & Plaut, 1987).”
Neural Programmer-Interpreters (2015, background): “Schmidhuber (1992) proposed a related meta-learning idea, whereby one learns the parameters of a slowly changing network, which in turn generates context dependent weight changes for a second rapidly changing network.”
Graph HyperNetworks for Neural Architecture Search (2018, background): “Schmidhuber (1992; 1993) proposed “fast-weights”, where one network produces weight changes for another.”
SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, methodology): “Our method is conceptually similar, but replaces random weights with weights generated through HyperNets [12], which are one of a class of techniques for dynamically adapting weights through use of an auxiliary model [9, 17, 27, 31].”
Meta Networks (2017, methodology): “Fast weights have also been used to implement recurrent nets (Schmidhuber, 1992).”
On the Binding Problem in Artificial Neural Networks (2020, background): “Alternatively, Bengio (2019) argues for the importance of a low-dimensional ‘conscious state’ (working memory) composed of largely independent units of abstraction that can be selected via attention (perhaps reminiscient of Schmidhuber, 1992b).”