Backpropagation Through Time: What It Does and How to Do It

By Paul J. Werbos. 1990

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Sequence to Sequence Learning with Neural Networks (2014, background): “In particular, while we were unable to train a standard RNN on the non-reversed translation problem (shown in fig.”
Character-Level Language Modeling with Deeper Self-Attention (2018, neutral): “This procedure is known as “truncated backpropagation through time” (TBTT), because the gradient computation doesn’t proceed further than a single batch [44].”
Cross-lingual Language Model Pretraining (2019, methodology): “In the case of LSTM language models, back-propagation through time [41] (BPTT) is performed by providing the LSTM with the last hidden state of the previous iteration.”
Improving Neural Language Models with a Continuous Cache (2016, methodology): “The gradient is computed using the truncated backpropagation through time algorithm (Werbos, 1990; Williams & Peng, 1990).”
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems (2015, methodology): “works were trained with back propagation through time (Werbos, 1990).”
Character-Aware Neural Language Models (2015, methodology): “The models are trained by truncated backpropagation through time (Werbos 1990; Graves 2013).”
Transfer Learning for Low-Resource Neural Machine Translation (2016, methodology): “The models were trained to optimize maximum likelihood (via a softmax layer) with back-propagation through time (Werbos, 1990).”
Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, methodology): “The core of BPTT algorithm is a technique called unrolling the RNN, such that we come up with a feed-forward network over time spans.”
Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, methodology): “We sed LSTM cells to build the decoded stacked architectures (as way to mitigate the exploding and vanishing gradient probems [14]), and Adam truncated through time [43] (i.”
Graph HyperNetworks for Neural Architecture Search (2018, methodology): “Similar to RNNs, GNNs are typically learned using backpropagation through time (BPTT) (Werbos, 1990).”
Forward and Reverse Gradient-Based Hyperparameter Optimization (2017, background): “…two classic alternative ways of computing gradients for recurrent neural networks (RNN) (Pearl-mutter, 1995): the Lagrangian (reverse) way corresponds to back-propagation through time (Werbos, 1990), while the forward way corresponds to real-time recurrent learning (RTRL) (Williams & Zipser, 1989).”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “Recurrent networks are dedicated sequence models that maintain a vector of hidden activations that are propagated through time (Elman, 1990; Werbos, 1990; Graves, 2012).”
A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction (2017, background): “Recurrent neural networks (RNNs) [Rumelhart et al., 1986; Werbos, 1990; Elman, 1991], a type of deep neural network specially designed for sequence modeling, have received a great amount of attention due to their flexibility in capturing nonlinear relationships.”
Using deep learning to detect price change indications in financial markets (2017, methodology): “The most commonly used method to minimize the loss function defined in Equation (12) and learn the parameters W of the model is gradient descent [15]:”
Ask the GRU: Multi-task Learning for Deep Text Recommendations (2016, background): “This is particularly important for deep models such as RNNs, since these high-capacity models are prone to overﬁtting.”
DDSP: Differentiable Digital Signal Processing (2020, background): “…in the asymptotic limit (Hornik et al., 1989), but their practical success is largely due to the use of strong structural priors such as convolution (Le-Cun et al., 1989), recurrence (Sutskever et al., 2014; Williams & Zipser, 1990; Werbos, 1990), and self-attention (Vaswani et al., 2017).”