UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

By Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou and 1 more. 2020

428 citations recorded by the source corpus.

We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM). Given an input text with masked tokens, we rely on conventional masks to learn inter-relations between corrupted tokens and context via autoencoding, and pseudo masks to learn intra-relations between masked spans via partially autoregressive modeling. With well-designed position embeddings and self-attention masks, the context encodings are reused to avoid redundant computation. Moreover, conventional masks used for autoencoding provide global masking information, so that all the position embeddings are accessible in partially autoregressive language modeling. In addition, the two tasks pre-train a unified language model as a bidirectional encoder and a sequence-to-sequence decoder, respectively. Our experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks across several widely used benchmarks.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Adversarial Training for Large Neural Language Models (2020, background): “, 2018) and other transformer-based models (Liu et al., 2019c; Radford et al., 2018; Clark et al., 2020; Dong et al., 2019; Bao et al., 2020).”
PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation (2020, background): “UniLMs (Dong et al., 2019; Hangbo et al., 2020) fine-tune BERT with an ensemble of masks, some of which use only leftward context, allowing UniLMs to be used for generation tasks.”
BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “[DBK+20] also report that pixel-level auto-encoding performs worse, although it is the most straightforward translation of BERT from NLP to CV.”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, methodology): “We report the results of MINILM distilled from an in-house pre-trained Transformer model following UNILM (Dong et al., 2019; Bao et al., 2020) in the BERTBASE size.”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, background): “Pre-training with Transformer [45] backbone networks has substantially advanced the state of the art across natural language processing [34, 10, 28, 22, 11, 36, 1, 7, 8, 4–6, The approaches of vision-language pre-training can be divided into two categories.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, background): “Relative position bias In computing self-attention, we follow [45, 1, 29, 30] by including a relative position bias B ∈ RM2×M2 to each head in computing similarity:”
Knowledge Neurons in Pretrained Transformers (2021, background): “Large-scale pretrained Transformers (Devlin et al., 2019; Liu et al., 2019; Dong et al., 2019; Clark et al., 2020; Bao et al., 2020) are usually learned with a language modeling objective on large-scale corpora, such as Wikipedia, where exists oceans of factual knowledge.”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, methodology): “We also employ various pre-trained language generation models, including GPT-2 (Radford et al., 2019), UniLM (Dong et al., 2019), UniLM-v2 (Bao et al., 2020), BERT-Gen (Bao et al., 2020), BART (Lewis et al., 2019), and T5 (Raffel et al., 2019), to tackle this task and test their generative commonsen”
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, background): “• Secondly, instead of maximizing the expected log-likelihood of the hybrid sequence under a conventional casual mask or a prefix mask like VALL-E and UniLM (Bao et al., 2020), ELLA-V computes loss only on acoustic to-kens and special tokens EndOfPhone(EOP) and EndOfSentence(EOS) .”
M6: A Chinese Multimodal Pretrainer (2021, background): “The tremendous success of NLP pretraining, including BERT [6], GPT [2, 28, 29], and also some other related studies [1, 7, 18, 25, 47], inspires the research in cross-modal representation learning.”
Pre-trained models for natural language processing: A survey (2020, background): “UniLM [38, 8] extends the task of mask prediction on three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction.”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “Following [33, 2, 26], we add a relative positional bias B to each head when computing the attention score:”
GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2021, methodology): “UniLMv2 (Bao et al., 2020) adopts partially autoregressive modeling for generation tasks, along with the autoencoding objective for NLU tasks.”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, methodology): “Study on bias terms In NLP, while absolute position embedding is used in the original Transformer, the relative position bias approach is later proved beneficial [4, 41].”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “It has become a de facto trend to use a pretrained language model (Devlin et al., 2019; Dong et al., 2019; Yang et al., 2019b; Bao et al., 2020) for downstream NLP tasks.”
BARTScore: Evaluating Generated Text as Text Generation (2021, neutral): “Abstractive Systems A1: Ptr-Gen [62]; A2: Bottom-up [17]; A3: Fast-Abs-RL [5]; A4: Two-stage-RL [74]; A5: BERT-Ext-Abs [41]; A6: BERT-Abs [41]; A7: Trans-Abs [41]; A8: UniLM-1 [10]; A9: UniLM-2 [2]; A10: T5-base [56]; A11: T5-large [56]; A12: T5-11B [56]; A13: BART [33]; A14: SemSim [42].”