Architectural Complexity Measures of Recurrent Neural Networks

By Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdinov, Yoshua Bengio. 2016

164 citations recorded by the source corpus.

In this paper, we systematically analyze the connecting architectures of recurrent neural networks (RNNs). Our main contribution is twofold: first, we present a rigorous graph-theoretic framework describing the connecting architectures of RNNs in general. Second, we propose three architecture complexity measures of RNNs: (a) the recurrent depth, which captures the RNN's over-time nonlinear complexity, (b) the feedforward depth, which captures the local input-output nonlinearity (similar to the "depth" in feedforward neural networks (FNNs)), and (c) the recurrent skip coefficient which captures how rapidly the information propagates over time. We rigorously prove each measure's existence and computability. Our experimental results show that RNNs might benefit from larger recurrent depth and feedforward depth. We further demonstrate that increasing recurrent skip coefficient offers performance boosts on long term dependency problems.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Character-Level Language Modeling with Deeper Self-Attention (2018, neutral): “o zero”. Character sequences not in the range [a-zA-Z] are converted to a single space. Finally, the text is lowercased. The size of the corpus is 100M characters. Following Mikolov et al. (2012) and Zhang et al. (2016), we split the data into 90M characters for train, 5M characters for dev, and 5M ”
Recurrent Batch Normalization (2016, methodology): “We follow Mikolov et al. (2012); Zhang et al. (2016) and use the ﬁrst 90M characters for training, the next 5M for validation and the ﬁnal 5M characters for testing.”
Contextual Sequence Modeling for Recommendation with Recurrent Neural Networks (2017, background): “Some of recently proposed RNN architectures use recurrent depth [23], which is a length of path between recurrent steps.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “These domains are dominated by recurrent architectures, with many specialized designs developed for these tasks (Zhang et al., 2016; Ha et al., 2017; Krueger et al., 2017; Grave et al., 2017; Greff et al., 2017; Merity et al., 2017).”
On Multiplicative Integration with Recurrent Neural Networks (2016, background): “Recently there has been a resurgence of new structural designs for recurrent neural networks (RNNs) [1, 2, 3].”
Multiplicative LSTM for sequence modelling (2016, methodology): “The results are shown in architecture test set error (bits/char) small LSTM (Zhang et al., 2016) 1.65 small LSTM (ours) 1.64 small deep LSTM (best) (Zhang et al., 2016) 1.63 small mLSTM 1.59 mRNN (Mikolov et al., 2012) 1.54 multiplicative integration RNN (Wu et al., 2016) 1.52 skipping RNN…”
Image Super-Resolution via Dual-State Recurrent Networks (2018, background): “Interested readers are referred to [47] for detailed information on this general formulation of a RNN.”
Detecting Cyber Attacks in Industrial Control Systems Using Convolutional Neural Networks (2018, background): “in [25] and since then by many others [47] [46] [21]).”
Discrete Flows: Invertible Generative Models of Discrete Data (2019, methodology): “We also evaluated on text8, using the preprocessing of Mikolov et al. (2012); Zhang et al. (2016) with 100M characters and a vocabulary size of K = 27 .”
Recurrent Neural Networks (2018, unknown): “, 2017) or recurrent skip coefficients (Zhang et al., 2016).”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “MSDNets innovative combined multi-scale feature maps [254] and dense connectivity [103] to enable the accurate early exit whilemaintaining higher accuracy.”
Representation Mixing for TTS Synthesis (2018, unknown): “Subsequent LSTM decode layers are conditioned on pre-net activations, attention activations, and the hidden state of the layer before, forming a series of skip connections [19, 20].”
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (2017, unknown): “, 2017) or recurrent skip coefficients (Zhang et al., 2016).”
Short-term stock market price trend prediction using a comprehensive deep learning system (2020, methodology): “In the previous work [46], the author performed an analysis of the RNN architecture complexity.”