Gradient-based learning algorithms for recurrent networks and their computational complexity

By Ronald J. Williams, David Zipser. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Generating Sequences With Recurrent Neural Networks (2013, background): “1, and the network can then be trained with gradient descent.”
Adaptive Computation Time for Recurrent Neural Networks (2016, background): “We expect this to be particularly beneficial when long sequences of halting decisions must be made, since each decision is likely to affect all subsequent ones, and sampling noise will rapidly accumulate (as observed for policy gradient methods [36]).”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “The model presented above is continuous and can thus be trained with stochastic gradient descent (SGD) method and back-propagation through time (Rumelhart et al., 1985; Williams & Zipser, 1995; Werbos, 1988).”
Sequence Transduction with Recurrent Neural Networks (2012, methodology): “(13) ∂ Pr(k′|t, u) ∂fkt = ∂ Pr(k′|t, u) ∂gku =Pr(k′|t, u) [δkk′−Pr(k|t, u)] The gradient with respect to the network weights can then be calculated by applying Backpropagation Through Time (Williams & Zipser, 1995) to each network independently.”
Learning Longer Memory in Recurrent Neural Networks (2014, neutral): “This allowed to efﬁciently train these models on large datasets by using only simple tools such as stochastic gradient descent and back-propagation through time (Williams & Zipser, 1995; Werbos, 1988).”
Recurrent Environment Simulators (2017, methodology): “This approach corresponds to a form of truncated backpropagation through time (Williams & Zipser, 1995) – the extreme of this strategy (with T equal to the length of the whole training sequence) was used by Zaremba et al. (2014).”
Forecasting Across Time Series Databases using Long Short-Term Memory Networks on Groups of Similar Series (2017, methodology): “s, softmax is heavily used in the Natural Language Processing (NLP) context to predict a categorical output o t for a given sequence at time step t. RNNs often use backpropagation through time (BPTT; Williams and Zipser, 1995) and real time recurrent learning (RTRL; Robinson and Fallside, 1987) as t”
PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem (2011, background): “, [59, 61, 27, 32, 12]) are general c omputers that allow for both sequential and parallel computations, unlike the strictly sequential FORTH-like language of Section 4.”
Learning Precise Timing with LSTM Recurrent Networks (2003, methodology): “…and more), while traditional RNNs with more costly update algorithms such as BPTT (Williams and Peng, 1990), RTRL (Robinson and Fallside, 1987, Williams and Zipser, 1992), or combinations thereof (Schmidhuber, 1992, Williams and Zipser, 1992), already fail to learn in the presence of 10-step…”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, neutral): “Now a gradient descent technique [299, 317, 208](see Sec.”
One Big Net For Everything (2018, background): “Nevertheless, this is often much harder than imitating teachers through gradient-based supervised learning [99, 105, 47][76] for LSTM [24, 12, 19].”
Spatio-temporal video autoencoder with differentiable memory (2015, background): “Motivated by these two shortcomings, we focus on reducing the supervision effort required to train recurrent neural networks, which are known for their ability to handle sequential input data Williams & Zipser (1995); Hochreiter et al. (2000).”
Learning to Forget: Continual Prediction with LSTM (2000, methodology): “Gradient-based methods– back-propagation through time (Williams & Zipser, 1992; Werbos, 1988) or real-time recurrent learning (Robinson & Fallside, 1987; Williams & Zipser, 1992) and their combination (Schmidhuber, 1992)—share an important limitation.”
Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes (2008, methodology): “More complex p such as adaptive recurrent neural networks (RNN) [115, 120, 55, 62, 47, 26, 93, 77, 78] will use a nonlinear mapping and possibly the entire history h ( ≤ t ) as a basis for the predictions.”
A First Look at Music Composition using LSTM Recurrent Neural Networks (2002, methodology): “In the " CONCERT " model, Mozer used sophisticated RNN procedures including BPTT, log-likelihood objective functions and probabilistic interpretation of the output values.”
Forecasting across time series databases using recurrent neural networks on groups of similar series: A clustering approach (2017, methodology): “…in the Natural Language Processing (NLP) context to predict a categorical output ot for a given sequence at time step t. RNNs often use backpropagation through time (BPTT; Williams and Zipser, 1995) and real time recurrent learning (RTRL; Robinson and Fallside, 1987) as the learning algorithms.”