Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies
By Robert S. DiPietro, Christian Rupprecht, Nassir Navab, Gregory Hager. 2017
26 citations recorded by the source corpus.
Recurrent neural networks (RNNs) have achieved state-of-the-art performance on many diverse tasks, from machine translation to surgical activity recognition, yet training RNNs to capture long-term dependencies remains difficult. To date, the vast majority of successful RNN architectures alleviate this problem using nearly-additive connections between states, as introduced by long short-term memory (LSTM). We take an orthogonal approach and introduce MIST RNNs, a NARX RNN architecture that allows direct connections from the very distant past. We show that MIST RNNs 1) exhibit superior vanishing-gradient properties in comparison to LSTM and previously-proposed NARX RNNs; 2) are far more efficient than previously-proposed NARX RNN architectures, requiring even fewer computations than LSTM; and 3) improve performance substantially over LSTM and Clockwork RNNs on tasks requiring very long-term dependencies.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dhruv Madeka
- Balakrishnan Narayanaswamy
- Kari Torkkola
- Ruofeng Wen
- Jan Gasthaus
- Bryan Lim
- Stefan Zohren
- Bernie Wang
In Computer Vision and Pattern Recognition
Citations in the corpus
- A Multi-Horizon Quantile Recurrent Forecaster (2017, background): “Another not as well-known type of recurrent net to solve the same long dependency issue is NARX RNN (DiPietro et al, 2017), which computes hidden state ht not only based on ht−1, but also a specific set of other past states, e.g. (ht−2, · · · , ht−D).”