MPNet: Masked and Permuted Pre-training for Language Understanding

By Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu. 2020

1,779 citations recorded by the source corpus.

BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models. Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for pre-training to address this problem. We argue that XLNet does not leverage the full position information of a sentence and thus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel pre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the dependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position information as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in XLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of down-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large margin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g., BERT, XLNet, RoBERTa) under the same model setting. We release the code and pre-trained model in GitHub\footnote{\url{this https URL}}.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Pre-Trained Models: Past, Present and Future (2021, background): “An important follower of permutation language modeling is MPNet (Song et al., 2020),”
Large Language Models on Graphs: A Comprehensive Survey (2023, methodology): “Along this line of approaches, ConGrat [53] adopts GAT [86] as the graph encoder and tries MPNet [34] as the language model encoder.”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, methodology): “To ensure the diversity and representativeness of topic distribution in selected prompts, we enforce a graph-based selecting algorithm based on prompt similarity produced by language models [50, 40, 48].”
MTEB: Massive Text Embedding Benchmark (2022, methodology): “Classification & STS On multilingual classification and STS, the multilingual MPNet provides the overall strongest performance.”
BERTopic: Neural topic modeling with a class-based TF-IDF procedure (2022, methodology): “Although BERTopic demonstrates competitive topic diversity scores, it is consistently out-performed by CTM.”
Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework (2021, methodology): “Like BERT, MPNet uses a transformer architecture, but with a permuted instead of a masked language modeling pre-training task (Song et al., 2020).”
TransGAN: Two Transformers Can Make One Strong GAN (2021, background): “Transformers in the NLP domain beneﬁt from multiple pre-training tasks (Devlin et al., 2018; Song et al., 2020).”
What Matters in Language Conditioned Robotic Imitation Learning Over Unstructured Data (2022, methodology): “We compare the lightweight paraphrase-MiniLM-L3-v2 language embeddings from our full model against several popular alternatives, such as the larger BERT [35], Distilroberta [34] and MPNet [33], which double the embedding size from 384 to 768.”
ReCode: Robustness Evaluation of Code Generation Models (2022, methodology): “To measure the similarity scores for the docstring perturbations, we first extract docstrings from each pair of perturbed and unperturbed data points, and we use sentence transformer all-mpnet-base-v2 (Song et al., 2020) to predict an embedding vector for each docstring.”
Entailment Tree Explanations via Iterative Retrieval-Generation Reasoner (2022, background): “More recently, large deep learning language models (black-box methods) have gained popularity (Song et al., 2020; Raffel et al., 2020), but their improvements in result quality came with a cost: the system’s outputs lack explainability and inspectability.”
Rethinking with Retrieval: Faithful Large Language Model Inference (2022, methodology): “Subsequently, we use the pre-trained MPNet model (Song et al., 2020) to select the most similar paragraph based on the cosine similarity between the sentence embeddings of the retrieved paragraph and the sentence.”
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (2023, methodology): “The index was built us-ing MPNET (Song et al., 2020), a popular semantic embedding model based on SentenceTrans-former (Reimers & Gurevych, 2019).”
OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs (2021, methodology): “The text features of WikiKG90Mv2 are obtained by using MPNet (Reimers and Gurevych, 2019; Song et al., 2020), which is shown to be significantly better sentence encoder (Reimers and Gurevych, 2019).”
ExpeL: LLM Agents Are Experiential Learners (2023, methodology): “Concretely, we used the Faiss vectorstore (Johnson, Douze, and J´egou 2019) as the experience pool, kNN re-triever and all-mpnet-base-v2 (Song et al. 2020) em-bedder to obtain top-k successful trajectories that have the maximum inner-product task similarity with the evaluation task.”
Improve Transformer Models with Better Relative Position Embeddings (2020, methodology): “The relative position embedding by Shaw et al. (2018) is proven to be effective and it is validated in BERT variants model training (Raffel et al., 2019; Song et al., 2020).”
Small-Bench NLP: Benchmark for small single GPU trained models in Natural Language Processing (2021, neutral): “We are interested in training and adding the scores for the other publicly available language models like XLNet (Yang et al., 2020), MPNet (Song et al., 2020), etc.”