Transformer-XL: Attentive Language Models beyond a Fixed-Length Context
By Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, Ruslan Salakhutdinov. 2019
4,418 citations recorded by the source corpus.
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.
Subfields: Artificial Intelligence, Large Language Models, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Graham Neubig
- Yejin Choi
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Jianfeng Gao
- Mohit Bansal
- Noah A. Smith
- Furu Wei
- Michael Lewis
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Ruslan Salakhutdinov
- Alan L. Yuille
- Kyunghyun Cho
- Mohit Bansal
- Jianfeng Gao
- Yonatan Belinkov
Citations in the corpus
- XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “First, the inductive bias of relative encodings improves generalization [9].”
- LLaMA: Open and Efficient Foundation Language Models (2023, background): “More recently, transformer networks, based on self-attention, have led to important improvements, especially for capturing long range dependencies (Vaswani et al., 2017; Radford et al., 2018; Dai et al., 2019).”
- Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, methodology): “We train two TransformerXL (Dai et al., 2019) language models similar in scale to GPT-2 (Radford et al., 2019) on a medium-scale corpus of Wikipedia articles (i.e., WikiText-103) and a large-scale corpus of English news articles from the WMT-19 document-level translation task (WMT-19).5 We present d”
- A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “Transformer-XL is a recent language modeling architecture that is based on transformer self-attention (Vaswani et al., 2017), but modified to better learn dependencies beyond a fixed length by adding a segment-level recurrence mechanism.”
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “(2017); Baevski & Auli (2018); Dai et al. (2019) ) for different words is important.”
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, background): “GPT, GPT-2, and other autoregressive transformer language models (Dai et al., 2019; Yang et al., 2019) employ multi-layer transformer decoder architectures.”
- CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “Current state-of-the-art meth-ods (Dai et al., 2019; Radford et al., 2019) train a neural network with parameters θ to minimize the negative log-likelihood over a dataset D = { x 1 , . . . , x | D | } : Because language models learn p θ ( x i | x <i ) , a new ˜ x of length m can be generated by…”
- SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “, 2019) combines an autoregressive loss and the Transformer-XL (Dai et al., 2019) architecture with a more than an eight-fold increase in data to achieve current stateof-the-art results on multiple benchmarks.”
- One Epoch Is All You Need (2019, background): “…2018) 100 SPN (Menick & Kalchbrenner, 2018) Not reported BERT (Devlin et al., 2018) 40 Mesh Transformer (Shazeer et al., 2018) 10 Transformer-XL (Dai et al., 2019) Not reported GPT-2 (Radford et al., 2019) Not reported (20 or 100) Sparse Transformer (Child et al., 2019) 70 - 120 dataset for…”
- Unified Scaling Laws for Routed Language Models (2022, background): “Our Transformer [Vaswani et al., 2017] is based on the architecture in [Radford et al., 2019] with relative positional encodings [Dai et al., 2019].”
- Improving language models by retrieving from trillions of tokens (2021, methodology): “, 2019), with some minimal changes: we replace LayerNorm with RMSNorm (Zhang and Sennrich, 2019) and use relative position encodings (Dai et al., 2019).”
- Challenges in Detoxifying Language Models (2021, methodology): “Our standard LM is a TransformerXL model (Dai et al., 2019) trained on the C4 dataset (Raffel et al.”
- Compressive Transformers for Long-Range Sequence Modelling (2019, methodology): “We build on the ideas of the TransformerXL (Dai et al., 2019) which maintains a memory of past activations at each layer to preserve a longer history of context.”
- Deduplicating Training Data Makes Language Models Better (2021, methodology): “Table 5 shows that the presence of near-duplicates of the evaluation set in the train set has a significant impact on model perplexity for two standard models: Transformer-XL (Dai et al., 2019), which was trained on LM1B, and GROVER (Zellers et al., 2019), which was trained on RealNews.”
- Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “Recent advances in techniques for language generation (e.g., GPT (Radford et al., 2018), GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), TransformerXL (Dai et al., 2019), XLNet (Yang et al., 2019)) powered by Transformers (Vaswani et al., 2017) and an increasing repository of available dat”
- Big Bird: Transformers for Longer Sequences (2020, background): “Prominently, Dai et al. [21], Sukhbaatar et al. [83], Rae et al. [74] have proposed auto-regresive models that work well for left-to-right language modeling but suffer in tasks which require bidirectional context.”