A Fixed Size Storage O(n3) Time Complexity Learning Algorithm for Fully Recurrent Continually Running Networks

By J. Urgen Schmidhuber. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “1991: UL-based history compression through a deep stack of RNNs Aworking Very Deep Learner (Section 3) of 1991 (Schmidhuber, 1992b, 2013a) could perform credit assignment across hundreds of nonlinear operators or neural layers, by using unsupervised pretraining for a hierarchy of RNNs.”
PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem (2011, methodology): “P may itself be the set of weight matrices of a separate RNN called RNN2, computing tasks for RNN1, and modifications of RNN1, using techniques for network-modifying networks as described in previous work (Schmidhuber, 1992b, 1993a,b).”
Self-Delimiting Neural Networks (2012, methodology): “Supervised RNNs can be trained to map sequences of input patterns to desired output sequences by gradient descent and other methods [78, 81, 47, 52, 44, 31, 75].”
Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes (2008, background): “Early work [48, 49, 52] described a predictor based on a recurrent neural network [104, 109, 46, 53 , 38, 70] (in principle a rather powerful computational device, even by today’s machine learning standards), predicting sensor y inputs including reward signals from the entire history of previous in”
Gradient calculations for dynamic recurrent neural networks: a survey (1995, methodology): “This technique has been discovered independently a number of times [110], [111].”
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions (1998, background): “Since jf 0 v (netv (t q))j kF 0 (t q)kA f 0 max, the following inequality is obtained: @#v (t q) @#u (t) n (f 0 max)q kWvkx kWuT kx kWkq 2 A n (f 0 max kWkA)q : (14) This inequality results from kWvkx = kWevkx kWkA kevkx kWkA (15) and kWuT kx = WT eu x kWkA keukx kWkA ; (16) where ek is the unit vec”
Reducing the Ratio Between Learning Complexity and Number of Time Varying Variables in Fully Recurrent Nets (1993, background): “(1)It should be noted that there is a fixed-size storage algorithm (a hybrid between BPTT and RTRL) with Rtime = O(m(2)) (better than with RTRL, worse than with BPTT) and Rspace = O(m (2)) (much better than with BPTT, same as with RTRL) ([7][4]).”
Planning simple trajectories using neural subgoal generators (1993, methodology): “The derivation of the more complex algorithm for the recurrent architecture 2 is analoguous tothe derivation of conventional discrete time recurrent net algorithms (e.g. (Robinson and Fallside,1987), (Williams, 1989), (Williams and Zipser, in press), (Schmidhuber, 1992)).”
New results on recurrent network training: unifying the algorithms and accelerating convergence (2000, methodology): “This method is an on-line technique, and the complexity is . 5) The block update approach (BU) [ 18 ]‐[20]: This is an on-line approach that updates the weights every data points using some aspects of the FP and BTT methods.”
On-line learning algorithms for locally recurrent neural networks (1999, methodology): “The algorithm in [23] is an hybrid method.”