Generalization of back-propagation to recurrent neural networks.

By Fernando J. Pineda. 1987

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Gated Graph Sequence Neural Networks (2015, methodology): “Learning is done via the Almeida-Pineda algorithm (Almeida, 1990; Pineda, 1987), which works by running the propagation to convergence, and then computing gradients based upon the converged solution.”
The Graph Neural Network Model (2009, positive): “6 On the other hand, in our case, a more efficient approach is possible, based on the Almeida–Pineda algorithm [58], [59].”
Compressing Neural Networks with the Hashing Trick (2015, background): “While this work focuses on general, fully connected feed forward neural networks, the technique of HashedNets could naturally be extended to other kinds of neural networks, such as recurrent neural networks (Pineda, 1987) or others (Bishop, 1995).”
Deep learning in neural networks: An overview (2014, neutral): “, Atiya & Parlos, 2000; Baldi, 1995; Gherrity, 1989; Kremer & Kolen, 2001; Pearlmutter, 1989, 1995; Robinson & Fallside, 1987; Rohwer, 1989; Schmidhuber, 1992a; Werbos, 1988;Williams, 1989;Williams & Peng, 1990;Williams & Zipser, 1988, 1989a, 1989b), also for equilibrium RNNs (Almeida, 1987; Pineda,”
On the Binding Problem in Artificial Neural Networks (2020, background): “Attractor states were also found to occur naturally in RNNs, especially when using symmetric recurrent weights (Almeida, 1987; Pineda, 1987).”
Incremental Few-Shot Learning with Attention Attractor Networks (2018, methodology): “We also show how this regularizer can be optimized, using recurrent back-propagation (Liao et al., 2018; Almeida, 1987; Pineda, 1987) we show empirically that our proposed method can produce state-of-the-art results in incremental few-shot learning on the challenging mini -ImageNet (Vinyals et al.,…”
Achieving Human Parity in Conversational Speech Recognition (2016, background): “While the basic structures have been well known for a long period [21, 22, 23, 24, 25, 26, 27], it is only recently that they have dominated the field as the best models for speech recognition.”
The microsoft 2016 conversational speech recognition system (2016, background): “While the basic structures have been well known for a long period [1, 2, 3, 4, 5, 6, 7], it is only recently that they have emerged as the best models for speech recognition.”
A Comprehensive Survey on Graph Neural Networks (2019, methodology): “GNNs uses the AlmeidaPineda algorithm [80], [81] to train its model.”
Residual Flows for Invertible Generative Modeling (2019, neutral): “On the other hand, our approach shares some similarities with Recurrent Backpropagation (Almeida, 1987; Pineda, 1987; Liao et al., 2018), since both approaches leverage convergent dynamics to modify the derivatives.”
Toward an Integration of Deep Learning and Neuroscience (2016, methodology): “The above examples suggest that at least some such mechanisms could be understood from the perspective of optimizing cost functions.”
Long Short-Term Memory (1997, unknown): “This section focuses on recurrent nets with time-varying inputs (as opposed to nets with stationary inputs and fixed-point-based gradient calculations; e.g., Almeida, 1987; Pineda, 1987).”
On-line learning algorithms for locally recurrent neural networks (1999, methodology): “The name that we use, i.e., causal recursive backpropagation (not to be confused with recurrent backpropagation [ 51 ], [52]) was chosen to remember the dual nature of the algorithm: BPTT style formulas are used to backpropagate the error through the neurons and recursive computation of derivatives ”
Fast Exact Multiplication by the Hessian (1994, methodology): “The recurrent backpropagation algorithm (Almeida, 1987; Pineda, 1987) consists of a set of forward equations which relax to a solution for the gradient, xi = 9 j wjiyj (10)”