Capacity and Trainability in Recurrent Neural Networks

By Jasmine Collins, Jascha Narain Sohl-Dickstein, David Sussillo. 2016

215 citations recorded by the source corpus.

Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and per-unit capacity bounds with careful training, for a variety of tasks and stacking depths. They can store an amount of task information which is linear in the number of parameters, and is approximately 5 bits per parameter. They can additionally store approximately one real number from their input history per hidden unit. We further find that for several tasks it is the per-task parameter capacity bound that determines performance. These results suggest that many previous results comparing RNN architectures are driven primarily by differences in training effectiveness, rather than differences in capacity. Supporting this observation, we compare training difficulty for several architectures, and show that vanilla RNNs are far more difficult to train, yet have slightly higher capacity. Finally, we propose two novel RNN architectures, one of which is easier to train than the LSTM or GRU for deeply stacked architectures.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Deep Learning Scaling is Predictable, Empirically (2017, background): “…on the VC dimension of particular deep neural network models, including showing that recurrent neural network models have the same effective capacity if the optimization scheme is well-tuned and training runs long enough (Harvey et al. (2017); Dziugaite and Roy (2017); Collins et al. (2017)).”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Short-Term Memory (LSTM, Hochreiter and Schmidhuber (1997)) or Update Gate RNN (UGRNN, Collins et al. (2017)).”
Recurrent Neural Networks for Semantic Instance Segmentation (2017, background): “In order to improve the performance on long sequences, the memory of the model can be increased by adding more units to each ConvLSTM [41].”
On Network Design Spaces for Visual Recognition (2019, background): “In the absence of rigorous theoretical understanding of deep networks, it is imperative to perform large-scale empirical studies of deep networks to aid development [6, 3, 27].”
On the State of the Art of Evaluation in Neural Language Models (2017, background): “Motivated by recent results from (Collins et al., 2016), we compare models on the basis of the total number of trainable parameters as opposed to the number of hidden units.”
Efficient Neural Architecture Search via Parameter Sharing (2018, methodology): “Additionally, as Collins et al. (2017) have established that RNN models with more parameters can learn to store more information, we limit the size of our ENAS cell to 24 M parameters.”
Google Vizier: A Service for Black-Box Optimization (2017, methodology): “[6] used Vizier to perform hyperparameter tuning studies that collectively contained millions of trials for a research project investigating the capacity of different recurrent neural network architectures.”
The importance of better models in stochastic optimization (2019, neutral): “The paper [7] uses approximately 750000 CPU days in its parameter search.”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, background): “Recent work (Collins et al., 2016) provides empirical evidence that the beneﬁts of gating are mostly rooted in improved trainability rather than increased capacity or expressivity.”
Gradients are Not All You Need (2021, unknown): “LSTM’s [Hochreiter and Schmidhuber, 1997], GRU [Chung et al., 2014], UGRNN [Collins et al., 2016] are such modiﬁcations.”
Input Switched Affine Networks: An RNN Architecture Designed for Interpretability (2016, methodology): “One change from (Collins et al., 2016) is that we exchange the cross-entropy error with an L 2 error.”
Hallucinations in Neural Machine Translation (2018, background): “In parallel, there has also been work in understanding RNNs through the framework of dynamical systems [31, 29, 22, 26, 3] and understanding their capacity in language tasks [14].”
AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2018, background): “However, learning where to look next is difficult due to the huge search space and limited capacity of hidden states [1, 37] to remember input history.”
Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions (2019, background): “Therefore, to enable a fair comparison, in other research communities such as natural language processing, it is common practice to consider the total number of trainable parameters in the different models compared (Collins et al., 2016; Ji et al., 2016; Jagannatha and Yu, 2016).”
A Survey on Evolutionary Neural Architecture Search (2020, methodology): “For the time-series data processing, RNNs are more suitable, so the encoding space may contain cells including -RNN cell, LSTM [204], gated recurrent unit (GRU) [227], minimally gated unit (MGU) [228], and update-gated RNN (UGRNN) [229].”
Investigating recurrent neural network memory structures using neuro-evolution (2019, background): “This work introduced a new neuro-evolution algorithm, Evolutionary eXploration of Augmenting Memory Models (EXAMM), for evolving recurrent neural architectures by directly incorporating powerful memory cells such as the ∆ -RNN, MGU, GRU, LSTM and UGRNN units into the evolutionary process.”
Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections (2016, background): “RNNs, on the other hand, have been shown to encode approximately 5 bits of information per parameter (Collins et al. (2016)). And since the number of parameters of an RNN is proportional to the square size of the hidden state, the amount of information encoded per hidden state is O(n(2)), which is m”
A dataset and architecture for visual reasoning with a working memory (2018, background): “Indeed, a large constraint in training network models to perform generic Turing-complete operations is the ability to train systems that compute in time [39, 37].”