XLNet: Generalized Autoregressive Pretraining for Language Understanding
By Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, Quoc V. Le. Neural Information Processing Systems, 2019
9,461 citations recorded by the source corpus.
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.
Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing
View Paper: https://arxiv.org/abs/1906.08237
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Graham Neubig
- Quoc V. Le
- Yoshua Bengio
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Furu Wei
- Timo Schick
- Mohit Bansal
- Jianfeng Gao
- Hannaneh Hajishirzi
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Graham Neubig
- Jianfeng Gao
- Percy Liang
- Noah A. Smith
- Hannaneh Hajishirzi
Citations in the corpus
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Early results on transfer learning for NLP leveraged recurrent neural networks [Peters et al., 2018; Howard and Ruder, 2018], but it has recently become more common to use models based on the “Transformer” architecture [Vaswani et al., 2017].”
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “In particular, while both BERT (Devlin et al., 2019) and XL-Net (Yang et al., 2019) XLNet, while we use the same learning rate for all layers.”
- Scaling Laws for Neural Language Models (2020, background): “Deep learning has recently seen rapid progress in language modeling, with state of the art models [RNSS18, DCLT18, YDY + 19, LOG + 19, RSR + 19] approaching human-level performance on many specific tasks [WPN + 19], including the composition of coherent multi-paragraph prompted text samples [RWC +…”
- Experience Grounds Language (2020, background): “…progress in NLP. Performance peaks have been reached in tasks such as language modeling (Radford et al., 2019; Zellers et al., 2019c; Keskar et al., 2019) and span-selection question answering (Devlin et al., 2019a; Yang et al., 2019; Lan et al., 2019) through massive data and massive models.”
- StereoSet: Measuring stereotypical bias in pretrained language models (2020, methodology): “In this section, we evaluate popular pretrained language models such as BERT (Devlin et al., 2019), R O BERT A (Liu et al., 2019), XLN ET (Yang et al., 2019) and GPT2 (Radford et al., 2019) on Stere-oSet.”
- Adversarial Training for Large Neural Language Models (2020, background): “3% points in test accuracy over RoBERTaLARGE, outperforms XLNet (Yang et al., 2019) by 5.”
- TTTTTackling WinoGrande Schemas (2020, neutral): “Other than encoder-only pretrained transformer architectures [1, 3, 9], encoder–decoder style pretrained transformers [2, 5] have been proven to be effective in text generation tasks as well as comprehension tasks.”
- How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “Big, deep neural language models that have been pre-trained on unlabeled text have proven to be extremely performant when fine-tuned on down-stream Natural Language Processing (NLP) tasks (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Lan et al., 2019; Raffel et al., 2019).”
- Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “…that a customer gave to a restaurant on a 1 - 1 The way different segments are handled depends on the language model being used; they may e.g. be assigned different segment embeddings (Devlin et al., 2019), be separated by special tokens (Liu et al., 2019; Yang et al., 2019) or simply be ignored.”
- Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, background): “We use Transformer-based models (Vaswani et al., 2017) to learn the probability p ( x i | x 1: i − 1 ) , which has been demonstrated to scale to large self-supervised models with outstanding performance in generation quality and representation learning, including BERT (Devlin et al., 2018), GPT-2 (R”
- Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “In addition to BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019b), we also include XLNet (Yang et al., 2019) as an example of a strong, but different, model architecture.”
- SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “…learning (Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Song et al., 2019; Zhang et al., 2019; Yang et al., 2019), we propose several strategies that pre-train the encoder and de-coder before optimizing them jointly with the auto-encoding…”
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “We adapt these hyperparameters from Liu et al. (2019), Devlin et al. (2019), and Yang et al. (2019).”
- UNITER: Learning UNiversal Image-TExt Representations (2019, background): “Recently, pre-trained language models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), GPT2 (Radford et al., 2019), and XLNet (Yang et al., 2019) have shown great advances for NLP tasks.”
- Technical report on Conversational Question Answering (2019, methodology): “Pretrained Model Pretrained language models, such as GPT(Alec Radford and Sutskever., 2018), BERT(Devlin et al., 2018), XLNET(Yang et al., 2019) and RoBERTa(Liu et al., 2019), have brought significant performance gains to many NLP tasks, including machine reading comprehension.”