Dilated Recurrent Neural Networks

By Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael J. Witbrock, Mark Hasegawa-Johnson, Thomas S. Huang. 2017

351 citations recorded by the source corpus.

Learning with recurrent neural networks (RNNs) on long sequences is a notoriously difficult task. There are three major challenges: 1) complex dependencies, 2) vanishing and exploding gradients, and 3) efficient parallelization. In this paper, we introduce a simple yet effective RNN connection structure, the DilatedRNN, which simultaneously tackles all of these challenges. The proposed architecture is characterized by multi-resolution dilated recurrent skip connections and can be combined flexibly with diverse RNN cells. Moreover, the DilatedRNN reduces the number of parameters needed and enhances training efficiency significantly, while matching state-of-the-art performance (even with standard RNN cells) in tasks involving very long-term dependencies. To provide a theory-based quantification of the architecture's advantages, we introduce a memory capacity measure, the mean recurrent length, which is more suitable for RNNs with long skip connections than existing measures. We rigorously prove the advantages of the DilatedRNN over other recurrent neural architectures. The code for our method is publicly available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Other means of enabling neural networks to learn long-range dependencies in data that have been investigated include alternative loss functions [42] and model architectures [1, 8, 14, 19, 28, 29, 32].”
N-BEATS: Neural basis expansion analysis for interpretable time series forecasting (2019, background): “A few earlier works explored the combinations of recurrent neural networks with dilation, residual connections and attention [5, 17, 26].”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “…in an LSTM with convolutional layers to allow for additional structure in the recurrent layers; the Quasi-RNN model (Bradbury et al., 2017) that interleaves convolutional layers with simple recurrent layers; and the dilated RNN (Chang et al., 2017), which adds dilations to recurrent architectures.”
Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks (2019, neutral): “Our best Speaker-ID result compares favorably with some recent works on the same dataset, such as [42] and [43].”
Efficiently Modeling Long Sequences with Structured State Spaces (2021, neutral): “[4] 99.20 98.13 73.42 TCN [3] 99.0 97.2 - LSTM [17, 21] 98.9 95.11 63.01 r-LSTM [42] 98.4 95.2 72.2 Dilated GRU [5] 99.0 94.6 - Dilated RNN [5] 98.0 96.1 - IndRNN [25] 99.0 96.0 - expRNN [24] 98 Baselines.”
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, background): “…for arranging local and global RNN layers, or more general the hierarchical RNNs that perform sequence modeling in multiple time scales [14–19], the stacked DPRNN blocks iteratively and alternately perform the intra-and inter-chunk operations, which can be treated as an interleaved…”
HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, background): “[10] Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang.”
Simplified State Space Layers for Sequence Modeling (2022, background): “[9] Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang.”
A survey on long short-term memory networks for time series prediction (2021, background): “An approach to tackle the problem concerning long-term dependencies that aims to establish a memory within regular RNNs is described by [5].”
Hierarchical reinforcement learning for efficient exploration and transfer (2020, background): “Dilated recurrent neural networks (Chang et al., 2017) are designed to maintain constant information during a given time period, similar to the idea of remaining in a given region for multiple timesteps, and”
N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting (2022, methodology): “In Appendix H we demonstrate how adding them to a DilRNN improves its performance.”
A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting (2020, neutral): “In standard LSTMs and related cells, part of the input at a time t is the hidden state from step t − 1.”
Deep Learning for Time Series Anomaly Detection: A Survey (2022, neutral): “Bayesian networks help refine observations due to their adaptability and ease in calculating probabilities.”
Where to Go Next: Modeling Long- and Short-Term User Preferences for Point-of-Interest Recommendation (2020, methodology): “To alleviate such incontinuity of model inputs, dilated RNNs (Chang et al. 2017) introduce a fixed skip length on the inputs, e.”
Trellis Networks for Sequence Modeling (2018, methodology): “In the sequential CIFAR-10 task, these images are passed into the model one at each time step, flattended as in the MNIST tasks.”
Timeseries Anomaly Detection using Temporal Hierarchical One-Class Network (2020, methodology): “First, it uses the dilated recurrent neural network (RNN) [2] with skip connections to efﬁciently extract multi-scale temporal features from the timeseries.”
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers (2021, background): “The matrices in Appendix D.3 and Appendix E.2 (Theorems 8, 9 and 11) actually satisfy this stronger structure, so Corollary E.3 applies.”
Deep Learning for Anomaly Detection in Time-Series Data: Review, Analysis, and Guidelines (2021, methodology): “Meanwhile, the dilated RNN, as illustrated in [124, Fig.”
Multi-range Reasoning for Machine Comprehension (2018, methodology): “Finally, our model also draws inspiration from dilation, in particular dilated RNNs [3] and dilated convolutions [14], that intuitively help to model long-range dependencies.”