Fast-Slow Recurrent Neural Networks

By Asier Mujika, Florian Meier, Angelika Steger. 2017

81 citations recorded by the source corpus.

Processing sequential data of variable length is a major challenge in a wide range of applications, such as speech recognition, language modeling, generative image modeling and machine translation. Here, we address this challenge by proposing a novel recurrent neural network (RNN) architecture, the Fast-Slow RNN (FS-RNN). The FS-RNN incorporates the strengths of both multiscale RNNs and deep transition RNNs as it processes sequential data on different timescales and learns complex transition functions from one time step to the next. We evaluate the FS-RNN on two character level language modeling data sets, Penn Treebank and Hutter Prize Wikipedia, where we improve state of the art results to $1.19$ and $1.25$ bits-per-character (BPC), respectively. In addition, an ensemble of two FS-RNNs achieves $1.20$ BPC on Hutter Prize Wikipedia outperforming the best known compression algorithm with respect to the BPC measure. We also present an empirical investigation of the learning and network dynamics of the FS-RNN, which explains the improved performance compared to other RNN architectures. Our approach is general as any kind of RNN cell is a possible building block for the FS-RNN architecture, and thus can be flexibly applied to different tasks.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Character-Level Language Modeling with Deeper Self-Attention (2018, background): “Fast-Slow Recurrent Neural Networks [34] also achieved near SOTA by increasing the number of recurrent steps for each character processed.”
An Analysis of Neural Language Modeling at Multiple Scales (2018, methodology): “…1.219 14.4M NASCell (small) (Zoph & Le, 2016) 1.228 6.6M NASCell (Zoph & Le, 2016) 1.214 16.3M FS-LSTM-2 (Mujika et al., 2017) 1.190 7.2M FS-LSTM-4 (Mujika et al., 2017) 1.193 6.5M 6 layer QRNN ( h = 1024 ) (ours) 1.187 13.8M 3 layer LSTM ( h = 1000 ) (ours) 1.175 13.8M (Ha et al., 2016) 1.340 27M…”
Augmenting Self-attention with Persistent Memory (2019, neutral): “[30] – Large FS-LSTM-4 47M 1.”
Dynamic Evaluation of Transformer Language Models (2019, neutral): “…test Hyper LSTM (Ha et al., 2017) 25M 1.34 HM-LSTM (Chung et al., 2017) 35M 1.32 Recurrent highway networks (Zilly et al., 2017) 46M 1.27 FS-LSTM (Mujika et al., 2017) 47M 1.25 awd-LSTM (Merity et al., 2018) 47M 1.23 Transformer + aux losses (Al-Rfou et al., 2018) 235M 1.06 Multiplicative LSTM…”
Meta-Learning with Warped Gradient Descent (2019, methodology): “Our implementation of a Warp-RNN can be seen as meta-learning “slow” weights to facilitate learning of “fast” weights [43, 32].”
Simple Recurrent Units for Highly Parallelizable Recurrence (2017, unknown): “27 FS-LSTM (Mujika et al., 2017) 47m 4 100 - 1.”
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN (2018, unknown): “24 Fast-slow LSTM [37] 1.”
Trellis Networks for Sequence Modeling (2018, unknown): “TrellisNet sets a new state of the art on this dataset as well, with perplexity 29.19: about 7.6% better than the contemporaneous Generic TCN (Bai et al., 2018) 3.0M 1.31 Independently RNN (Li et al., 2018) 12.0M 1.23 Hyper LSTM (Ha et al., 2017) 14.4M 1.219 NAS Cell (Zoph & Le, 2017) 16.3M 1.214 Fa”
Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model (2020, unknown): “45 LSTM 1800 units (Mujika et al., 2017) 14M 1.”