An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories

By Ronald J. Williams, Jing Peng. 1990

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Exploring the Limits of Language Modeling (2016, neutral): “We hope that our encouraging results, which improved the best perplexity of a single model from 51.3 to 30.0 (whilst reducing the model size considerably), and set a new record with ensembles at 23.7, will enable rapid research and progress to advance Language Modeling.”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, background): “One possible solution is truncating backpropagation through time [68], so that all examples have the same sequence length during training [52].”
Deep learning in neural networks: An overview (2014, neutral): “…1995; Gherrity, 1989; Kremer & Kolen, 2001; Pearlmutter, 1989, 1995; Robinson & Fallside, 1987; Rohwer, 1989; Schmidhuber, 1992a; Werbos, 1988;Williams, 1989;Williams & Peng, 1990;Williams & Zipser, 1988, 1989a, 1989b), also for equilibrium RNNs (Almeida, 1987; Pineda, 1987) with stationary inputs.”
Dota 2 with Large Scale Deep Reinforcement Learning (2019, methodology): “We apply the Adam optimizer [20] using truncated backpropagation through time[21] over samples of 16 timesteps.”
Sequence-to-sequence neural net models for grapheme-to-phoneme conversion (2015, methodology): “To train these encoder and decoder networks, we used back-propagation through time (BPTT) [26,27], with the error signal originating in the decoder network.”
Learning quadrupedal locomotion over challenging terrain (2020, methodology): “The loss function for training a GRU student policy is deﬁned as To improve the performance and computational efﬁciency of the training, we have implemented Truncated Backpropagation Though Time (Truncated BPTT) [51].”
Incremental Few-Shot Learning with Attention Attractor Networks (2018, methodology): “the truncated BPTT [39] (T-BPTT) which optimizes for T steps of gradient-based optimization, and is commonly used in meta-learning problems.”
Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition (2014, methodology): “We use the truncated backpropagation through time (BPTT) learning algorithm to update the model parameters [16].”
Learning Precise Timing with LSTM Recurrent Networks (2003, positive): “Its architecture permits LSTM to bridge huge time lags between relevant input events (1000 steps and more), while traditional RNNs with more costly update algorithms such as BPTT (Williams and Peng, 1990), RTRL (Robinson and Fallside, 1987, Williams and Zipser, 1992), or combinations thereof (Schmid”
An Analysis of Neural Language Modeling at Multiple Scales (2018, background): “In addition to a potential accuracy beneﬁt, longer BPTT windows can improve GPU utilization.”
High-quality Speech Coding with Sample RNN (2018, methodology): “During training, we use a batch size of 24 and a sequence length of 6400 samples for truncated back propagation through time (TBPTT, [26, 27]) on a single GPU.”
Highway long short-term memory RNNS for distant speech recognition (2015, methodology): “Truncated BPTT is usually performed for parameters updating, therefore, only a small segment (e.g., 20 frames) of each sequence has to be packed into the minibatch.”
Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognition (2018, methodology): “Each BPTT segment contains 20 frames, and each mini-batch contains 80 utterances, since we find empirically that 80 utterances has similar performance to 40 utterances.”
Artificial neural network modeling of water table depth fluctuations (2001, methodology): “Here we speciﬁcally make use of a variant of temporal back propagation, namely, the back propagation through time (BPTT) variation proposed by Williams Peng [1990].”
Data-to-text Generation with Macro Planning (2021, methodology): “Furthermore, the text generation stage made use of truncated BPTT (Williams and Peng, 1990) with truncation length 100.”
Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images (2015, methodology): “The truncated backpropagation through time (BPTT) learning algorithm [38] is used for training.”