Recurrent Highway Networks

By Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, J. Urgen Schmidhuber. 2016

433 citations recorded by the source corpus.

Many sequential processing tasks require complex nonlinear transition functions from one step to the next. However, recurrent neural networks with 'deep' transition functions remain difficult to train, even when using Long Short-Term Memory (LSTM) networks. We introduce a novel theoretical analysis of recurrent networks based on Gersgorin's circle theorem that illuminates several modeling and optimization issues and improves our understanding of the LSTM cell. Based on this analysis we propose Recurrent Highway Networks, which extend the LSTM architecture to allow step-to-step transition depths larger than one. Several language modeling experiments demonstrate that the proposed architecture results in powerful and efficient models. On the Penn Treebank corpus, solely increasing the transition depth from 1 to 10 improves word-level perplexity from 90.6 to 65.4 using the same number of parameters. On the larger Wikipedia datasets for character prediction (text8 and enwik8), RHNs outperform all previous results and achieve an entropy of 1.27 bits per character.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, methodology): “Finally, by re-placing the LSTM with a recurrent highway network (Zilly et al., 2016), state of the art results are achieved when applying weight tying.”
Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “Much of the progress in this area has been made by mitigating the vanishing gradients problem [14] by architectures such as LSTMs [15], GRU [3], Recurrent Highway Networks [51], Uni-tary RNNs [1] and others.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…advances, including but not limited to devising novel architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al.,…”
Deep Learning Scaling is Predictable, Empirically (2017, neutral): “We also compare LSTMs against Recurrent Highway Networks (RHNs) described in Zilly et al. (2017).”
Beyond human-level accuracy: computational challenges in deep learning (2019, background): “Recurrent-highway networks (RHN) have been shown to provide low character perplexities for character LMs [41].”
Compressive Transformers for Long-Range Sequence Modelling (2019, neutral): “7L LSTM (Graves, 2013) 1.67 LN HyperNetworks Ha et al. (2016) 1.34 LN HM-LSTM Chung et al. (2016) 1.32 ByteNet (Kalchbrenner et al., 2016) 1.31 RHN Zilly et al. (2017) 1.27 mLSTM Krause et al. (2016) 1.24 64L Transf.”
Pointer Sentinel Mixture Models (2016, background): “Other work has improved language modeling performance by modifying the RNN architecture to better handle increased recurrence depth (Zilly et al., 2016).”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “This precedent led many to use character-level language modeling as a benchmark to evaluate neural architectures (Kalchbrenner et al., 2016; Chung et al., 2017; Ha et al., 2017; Zilly et al., 2017; Melis et al., 2018; Al-Rfou et al., 2019).”
Unbounded cache model for online language modeling with open vocabulary (2017, background): “lished with the recent emergence of neural networks [7, 45]. In particular, recurrent networks are now the standard approach, achieving state-of-the-art performances on several challenging benchmarks [31, 55]. 3.2 Recurrent networks. Recurrent networks are a special case of neural networks speciﬁcal”
HyperNetworks (2016, background): “ME n-gram (Mikolov et al., 2012) 1.37 Batch Norm LSTM (Cooijmans et al., 2016) 1.32 Recurrent Dropout LSTM (Semeniuta et al., 2016) 1.301 1.338 Zoneout RNN (Krueger et al., 2016) 1.27 HM-LSTM 3 (Chung et al., 2016) 1.27 (Sutskever et al., 2011) 1.60 GF-RNN (Chung et al., 2015) 1.58 20.0 M Grid-LSTM ”
Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling (2016, methodology): “…introduction in Mikolov et al. (2010), many improvements have been proposed for RNNLMs , including different dropout methods (Zaremba et al., 2014; Gal, 2015), novel recurrent units (Zilly et al., 2016), and use of pointer networks to complement the recurrent neural network (Merity et al., 2016).”
Regularizing and Optimizing LSTM Language Models (2017, methodology): “Our models outperform custom-built RNN cells and complex regularization strategies that preclude the possibility of using optimized libraries such as the NVIDIA cuDNN LSTM. Finally, we explore the use of a neural cache in conjunction with our proposed model and show that this further improves the pe”
On the State of the Art of Evaluation in Neural Language Models (2017, background): “Then, having carefully tuned and compared LSTM and Recurrent Highway Network (Zilly et al., 2016) based language models, we ﬁnd that well-regularised LSTMs outperform their more recent counterparts.”
Efficient Neural Architecture Search via Parameter Sharing (2018, methodology): “Additionally, we augment the simple transformations between nodes in the constructed recurrent cell with highway connections (Zilly et al., 2017).”
DARTS: Differentiable Architecture Search (2018, methodology): “Other settings are similar to ENAS, where each operation is enhanced with a highway bypass (Zilly et al., 2016) and the cell output is deﬁned as the average of all the intermediate nodes.”