Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

By Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, Yanbo Gao. 2018

824 citations recorded by the source corpus.

Recurrent neural networks (RNNs) have been widely used for processing sequential data. However, RNNs are commonly difficult to train due to the well-known gradient vanishing and exploding problems and hard to learn long-term patterns. Long short-term memory (LSTM) and gated recurrent unit (GRU) were developed to address these problems, but the use of hyperbolic tangent and the sigmoid action functions results in gradient decay over layers. Consequently, construction of an efficiently trainable deep network is challenging. In addition, all the neurons in an RNN layer are entangled together and their behaviour is hard to interpret. To address these problems, a new type of RNN, referred to as independently recurrent neural network (IndRNN), is proposed in this paper, where neurons in the same layer are independent of each other and they are connected across layers. We have shown that an IndRNN can be easily regulated to prevent the gradient exploding and vanishing problems while allowing the network to learn long-term dependencies. Moreover, an IndRNN can work with non-saturated activation functions such as relu (rectified linear unit) and be still trained robustly. Multiple IndRNNs can be stacked to construct a network that is deeper than the existing RNNs. Experimental results have shown that the proposed IndRNN is able to process very long sequences (over 5000 time steps), can be used to construct very deep networks (21 layers used in the experiment) and still be trained robustly. Better performances have been achieved on various tasks by using IndRNNs compared with the traditional RNN and LSTM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “, 2018) and others that modify the internal architecture of RNNs to ease the optimization (Wu et al., 2016; Li et al., 2018).”
Recurrent Independent Mechanisms (2019, methodology): “Separate Recurrent Models: EnTNet (Henaff et al., 2016) and IndRNN (Li et al., 2018) can be viewed as a set of separate recurrent models.”
Efficiently Modeling Long Sequences with Structured State Spaces (2021, neutral): “[4] 99.20 98.13 73.42 TCN [3] 99.0 97.2 - LSTM [17, 21] 98.9 95.11 63.01 r-LSTM [42] 98.4 95.2 72.2 Dilated GRU [5] 99.0 94.6 - Dilated RNN [5] 98.0 96.1 - IndRNN [25] 99.0 96.0 - expRNN [24] 98 Baselines.”
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, background): “This poses an additional challenge as conventional sequential modeling networks, including RNNs and 1-D CNNs, have difficulty on learning such long-term temporal dependency [12].”
HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, background): “[49] Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao.”
Skeleton-Based Action Recognition With Directed Graph Neural Networks (2019, methodology): “With the development of deep-learning-based methods, the data-driven methods have become the mainstream [7, 25, 20, 28, 37, 18, 19, 14, 13, 21, 17, 16, 34, 29, 3].”
Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition (2020, background): “Table 5 compares non-graph methods [23, 20], graph-based methods with spatial edges [18, 21, 33, 34, 32] and with spatialtemporal edges [8].”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Independently RNN (IndRNN) [48] is proposed to prevent the gradient explosion and vanishing problems without using additional gates.”
Trellis Networks for Sequence Modeling (2018, background): “TrellisNet sets a new state of the art on this dataset as well, with perplexity 29.19: about 7.6% better than the contemporaneous Generic TCN (Bai et al., 2018) 3.0M 1.31 Independently RNN (Li et al., 2018) 12.0M 1.23 Hyper LSTM (Ha et al., 2017) 14.4M 1.219 NAS Cell (Zoph & Le, 2017) 16.3M 1.214 Fa”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, background): “The main reasons for this computational benefit are that (a) taking powers of diagonal matrices is trivial (speeding up both training (see also (Li et al., 2018)) and inference), while exponentiating dense matrices is computationally expensive, and (b) while nonlinear recurrences must be computed se”
Machine Learning in Aerodynamic Shape Optimization (2022, background): “Recurrent Neural Networks RNN gets its name from a feedback loop that points a hidden neuron back to itself, constituting a recurrent structure [257, 258].”
DeepAffinity: Interpretable Deep Learning of Compound-Protein Affinity through Unified Recurrent and Convolutional Neural Networks (2018, methodology): “We used molecular data from three public datasets: labeled compound-protein binding data fromBindingDB (Liu et al. ,2006), compound data in the SMILES format from STITCH (Kuhn et al. , 2007) and protein amino-acid sequences from UniRef (Suzek et al. , 2014).”
Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, neutral): “X-sub X-view Lie Group [39] 50.1 52.8 STA-LSTM [36] 73.4 81.2 VA-LSTM [48] 79.2 87.7 ARRN-LSTM [16] 80.7 88.8 Ind-RNN [20] 81.8 88.0 2-Stream 3DCNN [21] 66.8 72.6 TCN [11] 74.3 83.1 ClipCNN+MTLN [9] 79.6 84.8 Synthesized CNN [27] 80.0 87.2 CNN+Motion+Trans [15] 83.2 88.8 ST-GCN [46] 81.5 88.3…”