Training RNNs as Fast as CNNs

By Tao Lei, Yu Zhang, Yoav Artzi. 2017

179 citations recorded by the source corpus.

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Weighted Transformer Network for Machine Translation (2017, background): “Variants of recurrent networks that use strided convolutions eschew the traditional time-step based computation (Kaiser & Bengio, 2016; Lei & Zhang, 2017; Bradbury et al., 2016; Gehring et al., 2016; 2017; Kalchbrenner et al., 2016).”
DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding (2017, neutral): “2015), (Kalchbrenner, Grefenstette, and Blunsom 2014), (Zhao, Lu, and Poupart 2015), (Lei and Zhang 2017).”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, background): “(Zhao et al., 2015) 83.6 (1.6) 90.4 (0.7) 92.2 (1.2) 91.1 (1.0) / / SRU (Lei & Zhang, 2017) 84.8 (1.3) 89.7 (1.1) 93.4 (0.8) 93.9 (0.6) 89.1 (0.3) / Wide CNNs (Lei & Zhang, 2017) 82.2 (2.2) 88.8 (1.2) 92.9 (0.7) 93.2 (0.5) 85.3 (0.4) / Bi-LSTM (Graves et al., 2013) 84.6 (1.6) 90.2 (0.9) 94.7 (0.7) 9”
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids (2018, methodology): “Speciﬁcally, they ﬁrst employed the ideas on fast training of RNNs (Lei & Zhang, 2017; Bradbury et al., 2017) to encode the shared information beforehand and reuse them along the propagation steps.”
RNN Architecture Learning with Sparse Regularization (2019, background): “As shown by Schwartz et al. (2018) and Peng et al. (2018), a variety of state-of-the-art neural architectures are rational (Lei et al., 2017; Bradbury et al., 2017; Foer-ster et al., 2017, inter alia ), so learning parameter-efﬁcient rational RNNs is of practical value.”
Fully Neural Network Based Speech Recognition on Mobile and Embedded Devices (2018, background): “Recently, quasi RNN (QRNN) and simple recurrent unit (SRU) were developed for the purpose of fast training and inference of very long sequences [10, 11].”
Propagation Networks for Model-Based Control Under Partial Observation (2018, background): “Hence, inspired by the ideas on fast RNNs training [24], [25], we propose to encode the shared information beforehand and reuse them along the propagation steps.”
A Clinically Applicable Approach to Continuous Prediction of Future Acute Kidney Injury (2019, unknown): “…architectures, specifically the long short-term memory (LSTM) 35 , update gate RNN (UGRNN) and Intersection RNN 36 , simple recurrent units (SRU) 37,38 , gated recurrent units (GRU) 39 , the Neural Turing Machine (NTM) 40 , memory-augmented neural network (MANN) 41 , the Differentiable Neural…”
Enhancing Transformer for End-to-end Speech-to-Text Translation (2019, background): “While LSTMs are known to require long trainig time (Lei et al., 2017; Di Gangi and Federico, 2018; Kalchbren-ner et al., 2016), Transformer reduces the training time by performing parallel computation along all the time steps, similarly to convolutional neural networks (CNNs).”
Multi-range Reasoning for Machine Comprehension (2018, methodology): “This is in similar spirit to QRNNs [2] and SRUs [19] which reduces computation cost by pre-learning the gating vectors.”
Dynamic Multimodal Instance Segmentation guided by natural language queries (2018, methodology): “Instead of using LSTMs as recurrent cells, we employ SRUs [6], which allow the LM to process the natural language queries more eﬃciently than when using LSTMs.”
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN (2018, methodology): “In the proposed IndRNN, the processing of the input (Wxt + b) is independent at different timesteps and can be implemented in parallel as in [3, 28].”