Simple Recurrent Units for Highly Parallelizable Recurrence

By Tao Lei, Yu Zhang, Sida I. Wang, Huijing Dai, Yoav Artzi. 2017

309 citations recorded by the source corpus.

Common recurrent neural architectures scale poorly due to the intrinsic difficulty in parallelizing their state computations. In this work, we propose the Simple Recurrent Unit (SRU), a light recurrent unit that balances model capacity and scalability. SRU is designed to provide expressive recurrence, enable highly parallelized implementation, and comes with careful initialization to facilitate training of deep models. We demonstrate the effectiveness of SRU on multiple NLP tasks. SRU achieves 5—9x speed-up over cuDNN-optimized LSTM on classification and question answering datasets, and delivers stronger results than LSTM and convolutional models. We also obtain an average of 0.7 BLEU improvement over the Transformer model (Vaswani et al., 2017) on translation by incorporating SRU into the architecture.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “To examine underspecification, we construct a predictor ensemble by training the model from 5 random seeds for each of three RNN cell types: Simple Recursive Units (SRU, Lei et al. (2018)), Long 4.”
Structured Pruning of Large Language Models (2019, methodology): “SRU results Following the practice of (Lei et al., 2018), we train a 6-layer SRU model using a batch size of 64 and an unroll length of 256.”
On the Relation between Position Information and Sentence Length in Neural Machine Translation (2019, methodology): “Lei et al. (2018) replace feed-forward layers by their proposed SRU which also incorporates relative position.”
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers (2021, background): “Notable examples of this type of model include the Quasi-RNN or QRNN [5] and the Simple Recurrent Unit (SRU) [33], which are among the most eﬀective models in practice.”
LSTM-Based Battery Remaining Useful Life Prediction With Multi-Channel Charging Profiles (2020, unknown): “The average MAPEs for all batteries are summarized in Table 7; the results are 1.84%, 1.51% and 2.13%, and 1.02% for MC-RNN, MC-GRU, MC-SRU, and MC-LSTM, respectively.”
Delving into Deep Imbalanced Regression (2021, unknown): “• Manifold-Mixup (M-M IXUP ) (Verma et al., 2019): M-M IXUP extends the idea of M IXUP from input space to the hidden representation space, where the linear interpolations are performed in (multiple) deep hidden layers.”
A Lightweight Recurrent Network for Sequence Modeling (2019, background): “This idea is extended to SRU (Lei et al., 2018) where a minimal recurrent component is strengthened via an external highway layer.”
RWKV: Reinventing RNNs for the Transformer Era (2023, methodology): “The element-wise W KV computation is time-dependent, but can be readily parallelized along the other two dimensions (Lei et al., 2018) Additionally, token shift is implemented as a simple offset in the temporal dimension at each block using PyTorch (Paszke et al., 2019) library as…”
RNN Architecture Learning with Sparse Regularization (2019, background): “As shown by Schwartz et al. (2018) and Peng et al. (2018), a variety of state-of-the-art neural architectures are rational (Lei et al., 2017; Bradbury et al., 2017; Foerster et al., 2017, inter alia), so learning parameterefficient rational RNNs is of practical value.”
Autoregressive Knowledge Distillation through Imitation Learning (2020, unknown): “In all tasks, we use a recurrent neural network, speciﬁcally SRU (Lei et al., 2017), as the student model.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Inspired by QRNN, simple recurrent unit (SRU) [50] replaces Wh t − 1 with a lightweight Hadamard product of a vector-format parameter and the previous cellular state, that is, V (cid:2) c t − 1 , then the computation of c t becomes independent and parallelizable.”
Language Models with Transformers (2019, result): “Our reasoning is analogous to that guiding the design of the SRU (simple recurrent unit) (Lei et al., 2018).”
Simplified State Space Layers for Sequence Modeling (2022, background): “[34] Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi.”
A Survey on Deep Learning for Multimodal Data Fusion (2020, background): “SRNN (Lei et al., 2018) A fast variant in which the light recurrence and highway network are proposed to improve the learning efficiency for a parallelized implementation.”