Learning State Space Trajectories in Recurrent Neural Networks

By Barak A. Pearlmutter. 1989

Many neural network learning procedures compute gradients of the errors on the output layer of units after they have settled to their final values. We describe a procedure for finding E/wij, where E is an error functional of the temporal trajectory of the states of a continuous recurrent network and wij are the weights of that network. Computing these quantities allows one to perform gradient descent in the weights to minimize E. Simulations in which networks are taught to move through limit cycles are shown. This type of recurrent network seems particularly suited for temporally continuous domains, such as signal processing, control, and speech.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (2016, background): “Although early work such as [25, 10, 4] has realized the resulting limitations and suggested continuous-time dynamical systems approaches towards RNNs, the great majority of modern RNN implementations uses fixed time steps.”
On supervised learning from sequential data with applications for speech regognition (1999, background): “Recurrent neural networks (RNNs) (Rumelhart et al., 1986; Giles et al., 1994; Pearlmutter, 1989; Robinson, 1994; Robinson et al., 1996) provide another alternative for incorporating temporal dynamics and are discussed in more detail in a later section.”
Bidirectional recurrent neural networks (1997, background): “Recurrent neural networks (RNN’s) [5], [8], [12], [13], [15] provide another alternative for incorporating temporal dynamics and are discussed in more detail in a later section.”
New results on recurrent network training: unifying the algorithms and accelerating convergence (2000, methodology): “This method is suitable for applications which need an on-line adaptation property.”
My First Deep Learning System of 1991 + Deep Learning Timeline 1962-2013 (2013, methodology): “…are adaptations and extensions of an over 40-year-old algorithm, namely, supervised efﬁ-cient backprop [59, 90] (TL 1.2) (compare [60, 53, 75, 12, 67]) or BPTT/RTRL for RNN, e.g., [94, 73, 91, 68, 80, 61] (exceptions include two 2011 contests specialised on transfer learning [35]—but compare [23]).”
Systematicity and specialization in semantics (1999, methodology): “Error derivatives were then calculated using a version of back-propagation through time adapted for continuous units [25].”
Long Short-Term Memory (1997, unknown): “1 We do not use the expression “time constant” in the differential sense, as Pearlmutter (1995) does.”
Learning Algorithms for Networks with Internal and External Feedback (1990, background): “This approach has been pursued by (Robinson and Fallside, 1987), (Werbos, 1988), (Pearlmutter, 1989), (Rumelhart et al., 1986), (Williams and Zipser, 1989), (Gherrity, 1989) and others in the case where there is only internal feedback.”
Deep Learning Meets SAR: Concepts, models, pitfalls, and perspectives (2020, background): “RNNs [35] are another major class of deep networks.”