Reducing the Ratio Between Learning Complexity and Number of Time Varying Variables in Fully Recurrent Nets

By J. Urgen Schmidhuber. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (2019, background): “This embedding is a Tensor-Product Representation (TPR; Smolensky, 1990) in an end-to-end-differentiable TPR system (Schlag & Schmidhuber, 2018; Schmidhuber, 1992) that learns “internal spotlights of attention” (Schmidhuber, 1993).”
Meta Networks (2017, background): “The base and the meta-level models can be framed in a single learner (Schmidhuber, 1993) or in separate learners (Bengio et al., 1990; Hochreiter et al., 2001).”
Using Fast Weights to Attend to the Recent Past (2016, methodology): “Hinton and Plaut [1987] suggested that fast weights could be used to allow true recursion in a neural network and Schmidhuber [1993] pointed out that a system of this kind could be trained end-to-end using backpropagation, but neither of these papers actually implemented this method of achieving…”
Tagger: Deep Unsupervised Perceptual Grouping (2016, background): “For example, an RNN with fast weights [26] can rapidly associate or bind the patterns to which the RNN currently attends.”
PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem (2011, methodology): “P may itself be the set of weight matrices of a separate RNN call ed RNN2, computing tasks for RNN1, and modifications of RNN1, using techniques for network-modify ing networks as described in previous work [33, 35, 34].”
One Big Net For Everything (2018, methodology): “Many RNN-like models can be used to build general computers, e.g., RNNs controlling push-down automata [9, 41] or other types of differentiable memory [20] including differentiable fast weights [63, 65], as well as closely related RNN-based meta-learners [66, 25, 51].”
Random Feature Attention (2021, background): “There are interesting connections between gated R FA and fast weights (Schmidhuber, 1992; 1993; Ba et al., 2016; Miconi et al., 2018, inter alia ).”
Introducing Symmetries to Black Box Meta Reinforcement Learning (2021, methodology): “Black-Box Meta RL Black-box meta RL can be implemented by policies that receive the reward signal as input (Schmidhuber 1993b) and use memory to learn, such as recurrence in RNNs (Hochreiter, Younger, and Conwell 2001; Wang et al. 2016; Duan et al. 2016).”
Learning to Reason with Third-Order Tensor Products (2018, background): “Previous outer product-based Fast Weights [26], which share strong similarities to TPRs of order 2, have shown to be powerful associative memory mechanisms [31, 27].”
A Perspective on Objects and Systematic Generalization in Model-Based RL (2019, background): “Schmidhuber (1992; 1993); Schlag & Schmidhuber (2018)) it may be more easy to learn general functions of this kind.”
R-SQAIR: Relational Sequential Attend, Infer, Repeat (2019, background): “Existing approaches to unsupervised learning about objects and relations from visual data can be categorized into either parallel [11, 12, 10] or sequential [26, 25, 7, 18, 6, 5, 36], depending on the core mechanism responsible for inferring object representations from a single image.”
Linear Transformers Are Secretly Fast Weight Programmers (2021, methodology): “Ba et al. (2016) have already pointed out a relation between a variant of outer product-based FWPs (Schmidhuber, 1993) and attention (Bahdanau et al., 2015).”
Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity (2018, result): “Importantly, in contrast to other approaches using uniform plasticity (Schmidhuber, 1993a), including “fast weights” (Ba et al., 2016), the amount of plasticity in each connection (represented by α i,j ) is trainable , allowing the meta-optimizer to design complex learning strategies (see Miconi et…”