ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training

By Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou. 2020

501 citations recorded by the source corpus.

This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of optimizing one-step-ahead prediction in the traditional sequence-to-sequence model, the ProphetNet is optimized by n-step ahead prediction that predicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large-scale dataset (160GB), respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Better Fine-Tuning by Reducing Representational Collapse (2020, background): “ProphetNet improved over these numbers by introducing their own novel self-supervised task (Yan et al., 2020).”
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation (2020, methodology): “Following Yan et al. (2020), we set α0 = α1 = 1.”
Planning with Learned Entity Prompts for Abstractive Summarization (2021, neutral): “, 2020), ProphetNet (Qi et al., 2020), (Gliwa et al.”
Better & Faster Large Language Models via Multi-token Prediction (2024, background): “Multi-token prediction in language modelling Qi et al. (2020) argue that multi-token prediction encourages planning, improves representations and prevents the overfitting on local patterns that can result from teacher-forced training.”
Learning to Generate Questions by Learning to Recover Answer-containing Sentences (2021, methodology): “We use all official settings for UniLM, ProphetNet (Qi et al., 2020) and ELECTRA (Clark et al.”
Text Generation by Learning from Demonstrations (2020, unknown): “…which have inherent limitations) on CNN/DM: at the time of writing, our results (45.40/22.01/42.25 using GOLD-p or 44.82/22.09/41.81 using GOLD-s ) are higher than 44.17/21.47/41.11 (PEGASUS; Zhang et al., 2020) and 44.20/21.17/41.30 (ProphetNet; Yan et al., 2020), both slightly higher than BART.”
AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization (2021, background): “Many studies on the use of pre-trained language models in the abstractive summarization task (Liu and Lapata, 2019; Yan et al., 2020; Su et al., 2020; Yu et al., 2020) have been undertaken and have achieved the state-of-the-art performance.”
How well do you know your summarization datasets? (2021, unknown): “, 2019), (2) PROPHET (Qi et al., 2020) (Lewis et al.”
A Divide-and-Conquer Approach to the Summarization of Long Documents (2020, methodology): “Towards that directionavarietyofpre-trainingobjectiveshavebeensuggested thatarebettersuitedforthetaskofabstractivesummarization [7]–[9],[19].”
LightSeq: A High Performance Inference Library for Transformers (2021, background): “Sequence processing and generation have been fundamental capabilities for many natural language processing tasks, including machine translation, summarization, language modeling, etc (Luong et al., 2015; Qi et al., 2020; Dai et al., 2019).”
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization (2021, methodology): “, 2021) and ProphetNet (Qi et al., 2020) for comparison.”
XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages (2021, methodology): “Raffel et al. (2020); Qi et al. (2020) used fully pre-trained seq2seq models, while Zhang et al. (2020) introduced a summarization-speciﬁc pre-training objective to achieve state-of-the-art re-sults on multiple datasets.”
Learning by Semantic Similarity Makes Abstractive Summarization Better (2020, methodology): “Concurrent to our work, ProphetNet (Yan et al., 2020) and PEGASUS (Zhang et al.”
RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “…G EN is related to efforts in modeling non-local dependencies in generation, which include methods that predict multiple to-kens (Oord et al., 2018; Qi et al., 2020), rely on retrieval (Khandelwal et al., 2020), use bidirectional LMs (Serdyuk et al., 2018), employ contrastive learning (Su et al.,…”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, methodology): “978 ProphetNet 3 .”