Generalization of Back propagation to Recurrent and Higher Order Neural Networks

By Fernando J. Pineda. 1987

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yoshua Bengio
2. Sergey Levine
3. Ruslan Salakhutdinov
4. Trevor Darrell
5. Dacheng Tao
6. Quoc V. Le
7. Abhinav Kumar Gupta
8. Zhe Gan

Citations in the corpus

• Graph Normalizing Flows (2019, background): “A recent approach for memory saving based on recurrent backpropagation (RBP) [1, 20, 18] requires running message passing to convergence, followed by the approximate, iterative inversion of a large matrix.”
• Deep Equilibrium Models (2019, background): “Another thread of work on implicit layers traces back to some of the original papers on recurrent networks trained via recurrent backpropagation (RBP) [2, 38].”
• Machine Learning on Graphs: A Model and Comprehensive Taxonomy (2020, methodology): “This process is repeated several times and the GNN parameters Θ and Θ are learned with backpropagation via the Almeda-Pineda algorithm [6, 91].”
• Finding Structure in Time (1990, background): “Page 4 There are many ways in which this can be accomplished, and a number of interesting proposals have appeared in the literature (e.g. Jordan, 1986; Tank & Hopfield, 1987; Stornetta, Hogg, & Huberman, 1987; Watrous & Shastri, 1987; Waibel, Hanazawa, Hinton, Shikano, & Lang, 1987; Pineda, 1988; Wi”
• Mapping Part-Whole Hierarchies into Connectionist Networks (1990, unknown): “Pineda [22] describes an alternative to backpropagation”
• The Microsoft 2017 Conversational Speech Recognition System (2017, unknown): “The combination of deep networks and efficient training methods with older neural modeling concepts [2, 3, 4, 5, 6, 7, 8] have produced steady advances in both acoustic modeling [9, 10, 11, 12, 13, 14, 15] and language modeling [16, 17, 18, 19].”
• Multiscale Deep Equilibrium Models (2020, background): “Pineda [43] and Almeida [2] studied implicit differentiation techniques for training recurrent dynamics, also known as recurrent back-propagation (RBP) [37].”
• Accelerating the convergence of the back-propagation method (1988, background): “A few remarks on the relative merits of Newton-type methods, gradient, and conjugate direction [e.g. Parker (1987); Pineda (1987)] are appropriate at this point in order to set our suggested algorithm in a proper context with respect to other methods.”
• Lending direction to neural networks (1995, background): “The rst is a generalization of recurrent back-propagation (Pineda, 1987; Almeida, 1987) to a network of directional units.”
• Localist Attractor Networks (2001, background): “Attractor networks have a long history in neural network research, from relaxation models in vision (Marr & Poggio, 1976; Hummel & Zucker, 1983), to the early work of HopŽeld (1982, 1984), to the stochastic Boltzmann machine (Ackley, Hinton, & Sejnowski, 1985; Movellan & McClelland, 1993), to symmet”
• Generalization in Interactive Networks: The Benefits of Inhibitory Competition and Hebbian Learning (2001, methodology): “dress this concern and provide additional evidence for the idea that interactivity impairs generalization, networks using the Almeida-Pineda (AP) exed-point recurrent backpropagation algorithm (Almeida, 1987; Pineda, 1987a, 1987b, 1988 ) were also run.”
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network (2018, methodology): “sentially what is dubbed “deep networks” and “deep learning” today. 4 foundation of Back Propagation and used to train the weights and time delays of these networks to perform a wide variety of tasks [11, 25, 45, 46, 47, 48]. We will rely on Back Propagation Through Time for training the systems ana”
• Filter-enhanced MLP is All You Need for Sequential Recommendation (2022, methodology): “It is widely used as assistant modules to couple with CNN [4], RNN [36] and selfattention network [57] to construct deep models.”
• Biologically Plausible Error-Driven Learning Using Local Activation Differences: The Generalized Recirculation Algorithm (1996, methodology): “…with the cross-entropy error term (Hinton 1989a); the Almeida-Pineda (AP) algorithm for error backpropagation in a recurrent network (Almeida 1987; Pineda 1987a,b, 1988), and the contrastive Hebbian learning algorithm (CHL) used in the Boltzmann machine and deterministic variants (Ackley et al.…”
• A Proposal on Machine Learning via Dynamical Systems (2017, background): “We note that continuous dynamical systems approach was mentioned in [6–8].”