Higher Order Recurrent Neural Networks

By Rohollah Soltani, Hui Jiang. 2016

64 citations recorded by the source corpus.

In this paper, we study novel neural network structures to better model long term dependency in sequential data. We propose to use more memory units to keep track of more preceding states in recurrent neural networks (RNNs), which are all recurrently fed to the hidden layers as feedback through different weighted paths. By extending the popular recurrent structure in RNNs, we provide the models with better short-term memory mechanism to learn long term dependency in sequences. Analogous to digital filters in signal processing, we call these structures as higher order RNNs (HORNNs). Similar to RNNs, HORNNs can also be learned using the back-propagation through time method. HORNNs are generally applicable to a variety of sequence modelling tasks. In this work, we have examined HORNNs for the language modeling task using two popular data sets, namely the Penn Treebank (PTB) and English text8 data sets. Experimental results have shown that the proposed HORNNs yield the state-of-the-art performance on both data sets, significantly outperforming the regular RNNs as well as the popular LSTMs.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Full Resolution Image Compression with Recurrent Neural Networks (2016, methodology): “This idea parallels the work done in Higher Order RNNs [15], where linear connections are added between iterations, but not between the input and the output of the RNN.”
Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation (2016, methodology): “The most related models to ours are those proposed by Tai et al., (2015), Le et al., (2015), Kalchbrenner et al., (2015), Soltani and Jiang (2016), Ladhak et al., (2016).”
BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (2021, methodology): “These methods demonstrate improvements in different tasks including classification [16] and language modeling [25].”
Frustratingly Short Attention Spans in Neural Language Modeling (2017, methodology): “Furthermore, we reimplemented Higher Order Recurrent Neural Networks (HORNNs) by Soltani & Jiang (2016).”
Long-term Forecasting using Tensor-Train RNNs (2017, methodology): “We compared TT-RNN against 2 set of natural baselines: 1st-order RNN (vanilla RNN, LSTM), and matrix RNNs (vanilla MRNN, MLSTM), which use matrix products of multiple hidden states without factorization (Soltani & Jiang, 2016)).”
Dual Path Networks (2017, background): “For HORNNs, weights are shared across steps, i.e. ∀ t, k, .”