Language Modeling with Gated Convolutional Networks

By Yann Dauphin, Angela Fan, Michael Auli, David Grangier. 2016

2,954 citations recorded by the source corpus.

The pre-dominant approach to language modeling to date is based on recurrent neural networks. Their success on this task is often linked to their ability to capture unbounded context. In this paper we develop a finite context approach through stacked convolutions, which can be more efficient since they allow parallelization over sequential tokens. We propose a novel simplified gating mechanism that outperforms Oord et al (2016) and investigate the impact of key architectural decisions. The proposed approach achieves state-of-the-art on the WikiText-103 benchmark, even though it features long-term dependencies, as well as competitive results on the Google Billion Words benchmark. Our model reduces the latency to score a sentence by an order of magnitude compared to a recurrent baseline. To our knowledge, this is the first time a non-recurrent approach is competitive with strong recurrent models on these large scale language tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, methodology): “Our model is equipped with gated linear units (Dauphin et al., 2016) and residual connections (He et al., 2015a).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “Dauphin et al. (2017) showed that architectures using Gated Linear Units (GLU) that compute hidden representations as the element wise product of a convolution and sigmoid gate provide perplexities comparable to large LSTMs on large scale language modeling tasks.”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, neutral): “should confirm our findings on the recent new extended version of SNLI, MultiNLI(4) and should also investigate how the recently introduced gated convolutional architectures (Dauphin et al., 2016) perform in transfer scenarios.”
The Curious Case of Neural Text Degeneration (2019, background): “, 2012) and convolutional networks (Dauphin et al., 2017), the Transformer architecture (Vaswani et al.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al., 2018), improving regularization and…”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “, 2019), language modeling (Dauphin et al., 2017; Baevski and Auli, 2019), abstractive document summarization (Fan et al.”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “, 2016), gated convolutional networks (Dauphin et al., 2017) and self-attentional networks (Al-Rfou et al.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, methodology): “In the non-MoE Transformer feed-forward sub-layers, we replace the first linear projection and the activation function with the Gated Linear Unit (Dauphin et al., 2017; Shazeer, 2020), which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error L”
GLU Variants Improve Transformer (2020, background): “2 Gated Linear Units (GLU) and Variants [Dauphin et al., 2016] introduced Gated Linear Units (GLU), a neural network layer deﬁned as the component-wise product of two linear transformations of the input, one of which is sigmoid-activated.”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “We apply Gated Linear Units (GLU, v : R 2 d → R d , Dauphin et al. 2017) on the output of the convolution Y .”
Mixture-of-Experts with Expert Choice Routing (2022, methodology): “…sub-layers (only every other layers are MoE layers), we replace the ﬁrst linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation…”
Language Models as Knowledge Bases? (2019, methodology): “…compute h t given the word history, e.g. , by using a multi-layer perceptron (Bengio et al., 2003; Mikolov and Zweig, 2012), convolutional layers (Dauphin et al., 2017), recurrent neural networks (Zaremba et al., 2014; Merity et al., 2016; Melis et al., 2017) or self-attention mechanisms…”
Hierarchical Neural Story Generation (2018, methodology): “First, we generate the premise or prompt of the story using the convolutional language model from Dauphin et al. (2017).”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, background): “Each block consists of the following two layers connected in a residual fashion, with layer normalization (Ba et al., 2016) after each residual connection: where Gate refers to a 1 × 1 convolution that doubles the number of channels, followed by a gated linear unit (Dauphin et al., 2016).”
Free-Form Image Inpainting With Gated Convolution (2018, background): “Feature-wise gating has been explored widely in vision [12, 28, 39, 41], language [6], speech [27] and many other tasks.”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, methodology): “Inspired by [17], the convolution module starts with a gating mechanism [23]—a pointwise convolution and a gated linear unit (GLU).”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “Training: To handle the large vocabulary of Wikitext-103, we follow Dauphin et al. (2017) and Baevski & Auli (2018) in using adaptive softmax (Grave et al., 2016) and adaptive input for computational efficiency.”