Learning long-term dependencies in NARX recurrent neural networks

By Tsungnan Lin, Bill G. Horne, Peter Tino, C. Lee Giles. 1996

It has previously been shown that gradient-descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve long-term dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. We show that the long-term dependencies problem is lessened for a class of architectures called nonlinear autoregressive models with exogenous (NARX) recurrent neural networks, which have powerful representational capabilities. We have previously reported that gradient descent learning can be more effective in NARX networks than in recurrent neural network architectures that have "hidden states" on problems including grammatical inference and nonlinear system identification. Typically, the network converges much faster and generalizes better than other networks. The results in this paper are consistent with this phenomenon. We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventional recurrent neural networks. We show that although NARX networks do not circumvent the problem of long-term dependencies, they can greatly improve performance on long-term dependency problems. We also describe in detail some of the assumptions regarding what it means to latch information robustly and suggest possible ways to loosen these assumptions.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Memory Networks (2014, background): “MemNNs without time features also perform badly, as they have no notion of what is the most recent utterance, but still outperform RNNs on the difficulty 5 actor only task (although they do not completely solve the difficulty 1 task).”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “…both hierarchical and temporal representation has been among the long-standing challenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many temporal data (Schmidhuber, 1991; Mozer, 1993; El Hihi & Bengio, 1995; Lin et al., 1996; Koutník et al., 2014).”
A Clockwork RNN (2014, methodology): “One model that is similar in spirit to our approach is the NARX RNN 1 (Lin et al., 1996).”
Deep learning in neural networks: An overview (2014, background): “BP for weight-sharing feedforward NNs (FNNs) and recurrent NNs (RNNs) Using the notation of Section 2 for weight-sharing FNNs or RNNs, after an episode of activation spreading through differentiable ft , a single iteration of gradient descent through BP computes changes of all wi in proportion to…”
Deep Learning for Stock Market Prediction (2020, background): “The result of ht is the LSTM block’s output information at time t (Equation (10)) [45].”
Adaptive computation and machine learning (1998, methodology): “To limit this problem, a remedy was proposed motivated by recent studies [364] suggesting that the vanishing-gradients problem can be mitigated by the use of an explicit delay line applied to the output, which provides shorter paths for the effective propagation of error signals.”
Sequence Learning (2001, background): “, finding a set of weights consistent with the training data) is unsolvable [72]! Moreover, even if several recurrent architectures have the computational capability to represent arbitrary nonlinear dynamical systems, gradient-based training suffers long-term dependencies [11] [41], showing difficul”
Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability (2001, background): “6.1 Perspective 91 6.2 Introduction 91 6.3 Overview 92 6.4 Neural Networks and Polynomial Filters 92 6.5 Neural Networks and Nonlinear Adaptive Filters 95 6.6 Training Algorithms for Recurrent Neural Networks 101 6.7 Learning Strategies for a Neural Predictor/Identifier 101 6.7.1 Learning Strategies”
Advanced Methods for Time Series Prediction Using Recurrent Neural Networks (2011, background): “networks with FIR connections (El Hihi & Bengio, 1996), (Lin et al., 1996) and recurrent networks with both internal loops and feedback from the output (Parlos et al.”
Massive MIMO Systems for 5G and beyond Networks—Overview, Recent Trends, Challenges, and Future Research Direction (2020, background): “Thus, several architectures have been proposed recently to address this distant data problem in massive MIMO, such as long short-term memory (LSTM) and non-linear autoregressive network with exogenous inputs (NARX) [156,157].”
DeepCare: A Deep Dynamic Memory Model for Predictive Medicine (2016, methodology): “Many approaches have been proposed to solve the problem, such as Leaky Units [39], Nonlinear AutoRegressive models with eXogenous (NARX) [40] and Long-Short Term Memory (LSTM) [14].”
Neural networks for the prediction and forecasting of water resource variables: a review of modelling issues and applications (2000, background): “The inclusion of inputs at explicit time lags, resulting in NARX recurrent networks, has been found to considerably improve performance in such cases (Lin et al., 1996; Siegelmann et al., 1997).”
Fast-Slow Recurrent Neural Networks (2017, methodology): “For instance using RNN cells with good long-term memory, like EURNNs [21] or NARX RNNs [26, 8], for the Slow cell might boost the long-term memory of the FS-RNN”
Learning and Representing Temporal Knowledge in Recurrent Networks (2011, unknown): “We consider the nonlinear autoregressive exogenous (NARX) model [12], [13] and define a one-to-one correspondence between NARX and a fragment of temporal logic.”
BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (2021, background): “Higher-order propagation has been studied to improve gradient flow [16, 18, 25].”
Applying LSTM to Time Series Predictable through Time-Window Approaches (2000, background): “Neither are NARX networks [8] which allow for several distinct input time windows (possibly of size one) with di erent temporal o sets.”