Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

By Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, Ruslan Salakhutdinov. 2019

4,418 citations recorded by the source corpus.

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.

Subfields: Artificial Intelligence, Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Yoshua Bengio
  4. Luke S. Zettlemoyer
  5. Percy Liang
  6. Quoc V. Le
  7. Graham Neubig
  8. Yejin Choi

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Yejin Choi
  4. Jianfeng Gao
  5. Mohit Bansal
  6. Noah A. Smith
  7. Furu Wei
  8. Michael Lewis

In Model Architectures

  1. Yoshua Bengio
  2. Quoc V. Le
  3. Ruslan Salakhutdinov
  4. Alan L. Yuille
  5. Kyunghyun Cho
  6. Mohit Bansal
  7. Jianfeng Gao
  8. Yonatan Belinkov

Citations in the corpus