Gradient calculations for dynamic recurrent neural networks: a survey

By Barak A. Pearlmutter. 1995

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Neural Ordinary Differential Equations (2018, neutral): “The use of the adjoint method for training continuous-time neural networks was previously proposed (LeCun et al., 1988; Pearlmutter, 1995), though was not demonstrated practically.”
Deep learning in neural networks: An overview (2014, background): “GPUs excel at the fast matrix and vector multiplications required not only for convincing virtual realities but also for NN training, where they can speed up learning by a factor of 50 and more.”
Learning Precise Timing with LSTM Recurrent Networks (2003, background): “Typical RNN learning algorithms (Pearlmutter, 1995) perform gradient descent in a very general space of potentially noise-resistant algorithms using distributed, continuousvalued internal states to map real-valued input sequences to real-valued output sequences.”
Generative Temporal Models with Memory (2017, background): “Models that instead maintain information in large, variable-order histories, e.g., recurrent neural networks (Pearlmutter, 1995), can have signiﬁcant advantages over ones constrained by ﬁxed-order Markov assumptions.”
Learning to Forget: Continual Prediction with LSTM (2000, background): “This implies that the backpropagated error ( Pearlmutter, 1995 ) quickly either vanishes or blows up (Hochreiter & Schmidhuber, 1997; Bengio, Simard, & Frasconi, 1994).”
Unbiased Online Recurrent Optimization (2017, methodology): “This operation is referred to as tangent forward propagation Simard et al. (1991) and can also often be computed algebraically.”
LSTM recurrent networks learn simple context-free and context-sensitive languages (2001, background): “Until recently, however, standard RNNs [ 13 ] have been plagued by a major practical problem: the gradient of the total output error with respect to previous inputs quickly vanishes as the time lags between relevant inputs and errors increase [6], [7], [32].”
Advanced monitoring of machining operations (2010, background): “Further supervised NN approaches are also considered here due to their use in decision making during monitoring of machining: probabilistic NN (PNN) [185], recurrent NN (RNN) [186–188], artificial cellular NN (ACNN) [189], fuzzy logic NN (FLNN) or neurofuzzy systems (NFS) combiningNN and FLmethods t”
LSTM can Solve Hard Long Time Lag Problems (1996, background): “The following task, though, is more difficult than the above benchmark problems: it cannot be solved at all in reasonable time by RS (we tried various architectures) nor any other recurrent net learning algorithm we are aware of (see [13] for ·an overview).”
Experimental demonstration of reservoir computing on a silicon photonics chip (2014, result): “This is in contrast to traditional recurrent neural networks, which are notoriously difﬁcult to train 3,4 .”
Artificial neural network modeling of water table depth fluctuations (2001, methodology): “In practice, this method is considered as the best on-line technique for practical problems Pearlmutter , 1995] and has been widely used to train RNN to perform various tasks [ Williams and Zipser , 1995; Puskorius et 1996; Feldkamp and Puskorius , 1998; Coulibaly et al ., 2001]. comprehensive…”
Long Short-Term Memory (1997, unknown): “However, 1 We do not use the expression “time constant” in the differential sense, as Pearlmutter (1995) does. at later training stages, j may suddenly start to cause avoidable errors in situations that already seemed under control by attempting to participate in reducing more difficult…”
Modelling Gene Expression Data using Dynamic Bayesian Networks (2006, unknown): “Neural Networks literature [Pea95].”