Character-Level Language Modeling with Deeper Self-Attention

By Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, Llion Jones. 2018

421 citations recorded by the source corpus.

LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model (Vaswani et al. 2017) with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “(2017) is (L=6, H=1024, A=16) with 100M parameters for the encoder, and the largest Transformer we have found in the literature is (L=64, H=512, A=2) with 235M parameters (Al-Rfou et al., 2018).”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “It has recently also become common to use models consisting of a single Transformer layer stack, with varying forms of self-attention used to produce architectures appropriate for language modeling [Radford et al., 2018; Al-Rfou et al., 2019] or classification and span prediction tasks [Devlin et al”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “With a deep root in density estimation 4 [4, 32, 24], language modeling has been a rapidly-developing research area [9, 1, 3].”
End-to-End Object Detection with Transformers (2020, methodology): “We found helpful to use auxiliary losses [1] in decoder during training, especially to help the model output the correct number of objects of each class.”
Generating Long Sequences with Sparse Transformers (2019, background): “We found that Transformers were difﬁcult to train with many layers, as noted by (Al-Rfou et al., 2018).”
Cross-lingual Language Model Pretraining (2019, background): “In the case of Transformers, previous hidden states can be passed to the current batch [1] to provide context to the first words in the batch.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, methodology): “…et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al., 2018), improving regularization and optimization algorithms Zaremba et al. (2014); Inan et al. (2016); Press & Wolf (2016); Merity…”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “Recently, Al-Rfou et al. (2018) demonstrated that self-attentional models can perform very well on language modeling tasks where the input and output is both characters.”
Compressive Transformers for Long-Range Sequence Modelling (2019, neutral): “Al-Rfou et al. (2019) 1.06 24L TXL (Dai et al., 2019) 0.99 Sparse Transf.”
Reformer: The Efficient Transformer (2020, background): “, 2018) while the number of layers goes up to 64 in (Al-Rfou et al., 2018).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “, 2017) were shown to be more effective than recurrent neural networks (Radford et al., 2018; Al-Rfou et al., 2019; Kaplan et al., 2020).”
Longformer: The Long-Document Transformer (2020, neutral): “Dataset text8 T12 (Al-Rfou et al., 2018) 44M - 1.”
Learning Deep Transformer Models for Machine Translation (2019, methodology): “For example, Al-Rfou et al. (2018) trained a character language model with a 64- layer Transformer encoder by resorting to auxiliary losses in intermediate layers.”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, methodology): “While the original Transformer included a separate encoder that processes input text and a decoder that generates target text, most recent LLMs are causal decoder-only (CD) models trained to autoregressively predict a text sequence (Liu et al., 2018; Radford et al., 2018; Al-Rfou et al., 2019).”
ReZero is All You Need: Fast Convergence at Large Depth (2020, background): “Transformers gained signiﬁcant popularity and success both in supervised and unsupervised NLP tasks [23, 11].”
Adaptive Attention Span in Transformers (2019, methodology): “2 to models of Al-Rfou et al. (2019) and Dai et al. (2019). For small models, our model outperforms the other Transformers by 0.”
On Layer Normalization in the Transformer Architecture (2020, background): “This architecture has achieved state-of-the-art performance in many tasks including language modeling (Dai et al., 2019; Al-Rfou et al., 2018) and machine translation (Dehghani et al., 2018; Edunov et al., 2018).”
Understanding and Improving Layer Normalization (2019, methodology): “ of each head is 64. The dropout rate is 0.1. The batch size is 22. We use optimizer Adam with a learning rate 0.00025. We use the average number of Bits-Per-Character (BPC) as the evaluation metric [Al-Rfou et al., 2018, Dai et al., 2019]. Text classiﬁcation includes two sentence classiﬁcation data”