An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
By Shaojie Bai, J. Kolter, Vladlen Koltun. 2018
6,867 citations recorded by the source corpus.
For most deep learning practitioners, sequence modeling is synonymous with recurrent networks. Yet recent results indicate that convolutional architectures can outperform recurrent networks on tasks such as audio synthesis and machine translation. Given a new sequence modeling task or dataset, which architecture should one use? We conduct a systematic evaluation of generic convolutional and recurrent architectures for sequence modeling. The models are evaluated across a broad range of standard tasks that are commonly used to benchmark recurrent networks. Our results indicate that a simple convolutional architecture outperforms canonical recurrent networks such as LSTMs across a diverse range of tasks and datasets, while demonstrating longer effective memory. We conclude that the common association between sequence modeling and recurrent networks should be reconsidered, and convolutional networks should be regarded as a natural starting point for sequence modeling tasks. To assist related work, we have made code available at this http URL .
Subfields: Artificial Intelligence, Model Architectures, Evaluation & Benchmarks, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Sergey Levine
- Percy Liang
- Ruslan Salakhutdinov
- Jianfeng Gao
- Dacheng Tao
In Model Architectures
- Yoshua Bengio
- Ruslan Salakhutdinov
- Vladlen Koltun
- Christopher R'e
- Tri Dao
- J. Kolter
- Kyunghyun Cho
- Shaojie Bai
In Evaluation & Benchmarks
- Jitendra Malik
- Vladlen Koltun
- Philipp Krahenbuhl
- Stefan Roth
- Giovanni Maria Farinella
- Antonino Furnari
- Silvio Savarese
- Andrew Zisserman
In Signal Processing
Citations in the corpus
- Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, methodology): “For text, these correspond to n-gram models [43] and convolutional neural language models [44], [45].”
- Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting (2019, methodology): “There has been a concurrent work [25], where an RNN has been used to evolve a global state common to all time-series.”
- Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, methodology): “(TCN) models [29]–[31], Conv-TasNet uses stacked dilated 1D convolutional blocks to replace the deep LSTM networks for the separation step.”
- Learning quadrupedal locomotion over challenging terrain (2020, positive): “The student policy uses a temporal convolutional network (TCN) [22] encoder.”
- Deep Equilibrium Models (2019, background): “As shown in Table 1, DEQ demonstrates good memory retention over relatively long sequences ( T = 400 ), with substantially better results than recurrent architectures such as LSTM/GRU (consistent with the findings in [7]).”
- Efficiently Modeling Long Sequences with Structured State Spaces (2021, background): “[4] 99.20 98.13 73.42 TCN [3] 99.0 97.2 - LSTM [17, 21] 98.9 95.11 63.01 r-LSTM [42] 98.4 95.2 72.2 Dilated GRU [5] 99.0 94.6 - Dilated RNN [5] 98.0 96.1 - IndRNN [25] 99.0 96.0 - expRNN [24] 98 Baselines.”
- Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, methodology): “We also consider two standard sequence-to-sequence models: a single LSTM architecture [65], [66] (LSTM), and Temporal Convolutional Networks [72] (TCN).”
- HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, background): “Many modern solutions such as attention [70] and dilated convolutions [5], are functions on finite windows, thus sidestepping the need for an explicit memory representation.”
- SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Prediction (2021, methodology): “And the TCN cascade 4 layers.”