Quasi-Recurrent Neural Networks

By James Bradbury, Stephen Merity, Caiming Xiong, Richard Socher. 2016

474 citations recorded by the source corpus.

Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep's computation on the previous timestep's output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, background): “Recent work has applied convolutional neural networks to sequence modeling such as Bradbury et al. (2016) who introduce recurrent pooling between a succession of convolutional layers or Kalchbrenner et al. (2016) who tackle neural translation without attention.”
A Constructive Prediction of the Generalization Error Across Scales (2019, positive): “, 2010), WikiText-2 (Bradbury et al., 2017), and WikiText-103 (Merity et al.”
Compressive Transformers for Long-Range Sequence Modelling (2019, background): “Merity et al. (2016) proposed the WikiText-103 dataset, which contains text from a high quality subset of English-language wikipedia articles.”
Multi-Task Self-Supervised Learning for Robust Speech Recognition (2020, positive): “QRNN can learn long-term dependencies in a very efficient way using convolutional gates across the time steps and a minimal recurrent pooling function.”
Regularizing and Optimizing LSTM Language Models (2017, methodology): “This can be done either through restricting the capacity of the matrix (Arjovsky et al., 2016; Wisdom et al., 2016; Jing et al., 2016) or through element-wise interactions (Balduzzi & Ghifary, 2016; Bradbury et al., 2016; Seo et al., 2016).”
Deep Voice: Real-time Neural Text-to-Speech (2017, methodology): “We ﬁnd that our models perform better, train faster, and require fewer parameters if we instead ﬁrst encode the inputs with a stack of bidirectional quasi-RNN (QRNN) layers (Bradbury et al., 2016) and then perform upsampling by repetition to the desired frequency.”
Improving Neural Language Modeling via Adversarial Training (2019, methodology): “In this case, we use Quasi-Recurrent neural networks (QRNN)-based language models (Merity et al., 2018b; Bradbury et al., 2017) as our base model for ef-ﬁciency.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “Language modeling remains one of the primary applications of recurrent networks and many recent works have focused on optimizing LSTMs for this task (Krueger et al., 2017; Merity et al., 2017).”
Deep Voice 2: Multi-Speaker Neural Text-to-Speech (2017, methodology): “The Deep Voice 2 vocal model is based on a WaveNet architecture (Oord et al., 2016) with a two-layer bidirectional QRNN (Bradbury et al., 2017) conditioning network, similar to Deep Voice 1.”
Weighted Transformer Network for Machine Translation (2017, background): “Variants of recurrent networks that use strided convolutions eschew the traditional time-step based computation (Kaiser & Bengio, 2016; Lei & Zhang, 2017; Bradbury et al., 2016; Gehring et al., 2016; 2017; Kalchbrenner et al., 2016).”
Deep Equilibrium Models (2019, background): “Recurrent networks (RNNs) [21, 51] as well as their variants such as LSTM [26] are universally applied and optimized in a variety of time-series tasks [9, 22, 34].”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “e combinations (hundreds) due to the resources needed, such as some large models or some large training scenarios.Forthisreason,wealsorefrainedtoevaluateotherNLM architectures suchas LSTMs [45],QRNNs [16], Transformers[85], oradditionalneuralcachevariants[62,86].Forthesamereason,as in [41] we also l”
BP-Transformer: Modelling Long-Range Context via Binary Partitioning (2019, positive): “We use pre-trained GloVe embedding (Pennington et al., 2014) (Li et al., 2015) 49.8 - Tree-LSTM (Socher et al., 2013) 51.0 - QRNN (Bradbury et al., 2017) - 91.4 BCN+Char+CoVe (McCann et al., 2017) 53.7 91.8 Table 1: Test accuracy on SST-5 and IMDB.”
Simplified State Space Layers for Sequence Modeling (2022, background): “[7] James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, methodology): “ stacking many blocks of feedforward layers with multi-head selfattention models [104]. The parameters in all blocks shared across time similar to the time-delayed neural networks [105] and quasi-RNN [106] without an explicit temporal order. Compared with RNN based encoder-decoder models [27], it ca”