XLNet: Generalized Autoregressive Pretraining for Language Understanding

By Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, Quoc V. Le. Neural Information Processing Systems, 2019

9,461 citations recorded by the source corpus.

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

View Paper: https://arxiv.org/abs/1906.08237

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Yejin Choi
  6. Graham Neubig
  7. Quoc V. Le
  8. Yoshua Bengio

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Yejin Choi
  4. Furu Wei
  5. Timo Schick
  6. Mohit Bansal
  7. Jianfeng Gao
  8. Hannaneh Hajishirzi

In Natural Language Processing

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Yejin Choi
  4. Graham Neubig
  5. Jianfeng Gao
  6. Percy Liang
  7. Noah A. Smith
  8. Hannaneh Hajishirzi

Citations in the corpus