PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation

By Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang. 2020

78 citations recorded by the source corpus.

Self-supervised pre-training, such as BERT, MASS and BART, has emerged as a powerful technique for natural language understanding and generation. Existing pre-training techniques employ autoencoding and/or autoregressive objectives to train Transformer-based models by recovering original word tokens from corrupted text with some masked tokens. The training goals of existing techniques are often inconsistent with the goals of many language generation tasks, such as generative question answering and conversational response generation, for producing new text given context. This work presents PALM with a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus, specifically designed for generating new text conditioned on context. The new scheme alleviates the mismatch introduced by the existing denoising scheme between pre-training and fine-tuning where generation is more than reconstructing original text. An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Minlie Huang
3. Jie Tang
4. Maosong Sun
5. Xipeng Qiu
6. Jianfeng Gao
7. Yang Liu
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Zhiyuan Liu
2. Jie Tang
3. Ming Ding
4. Ji-rong Wen
5. Chenliang Li
6. Ming Yan
7. Songfang Huang
8. Wayne Xin Zhao

In Large Language Models

1. Jie Tang
2. Zhiyuan Liu
3. Minlie Huang
4. Xiao Liu
5. Ming Ding
6. Daniel S. Weld
7. Xu Han
8. Kyle Lo

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, methodology): “There are following works that specify in typical seq2seq tasks, such as PEGASUS (Zhang et al., 2020a) and PALM (Bi et al., 2020).”
• GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2021, methodology): “Encoder-decoder models adopt bidirectional attention for the encoder, unidirectional attention for the decoder, and cross attention between them (Song et al., 2019; Bi et al., 2020; Lewis et al., 2019).”
• mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022, methodology): “Prefix Language Modeling (PrefixLM): This task aims to generate the caption given an image and predict the text segment subsequent to the cross-modal context as [30].”
• VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation (2020, background): “(7) 4 , can be further formalized as: Several monolingual pre-training models such as MASS (Song et al., 2019), BART (Lewis et al., 2019) and PALM (Bi et al., 2020) also present similar unidirectional language modeling tasks.”
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2023, unknown): “5, GPT-4 [36], PaLM [2, 4], and BLOOM [38].”
• Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “This triggers a number of studies to explore the performance limit by training ever larger PLMs (e.g., the 175B GPT-3 [16] and the 540B PaLM [11]).”
• TLDR: Extreme Summarization of Scientific Documents (2020, background): “Additionally, various pretraining meth-ods of transformer models have been found to be successful in extreme summarization (Zhang et al., 2019; Lewis et al., 2019; Liu and Lapata, 2019; Bi et al., 2020).”
• All NLP Tasks Are Generation Tasks: A General Pretraining Framework (2021, unknown): “PALM (Bi et al., 2020) is pre-trained for generating coherent text from given context and adds a BERT-based autoencoding objective to the encoder.”
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Following MASS, a number of encoder-decoder models like T5 [6], mT5 [99], mT6 [91], BART [8], mBART [65], PLBART [37], PEGAUSUS [9] and PALM [156] are proposed in the recent times.”
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning (2021, background): “Self-supervised pre-training has achieved great success in a wide range of natural language understanding (Devlin et al., 2018; Liu et al., 2019; Wang et al., 2019; Lan et al., 2019) and generation tasks (Song et al., 2019; Lewis et al., 2019; Bi et al., 2020).”
• Addressing Semantic Drift in Generative Question Answering with Auxiliary Extraction (2021, background): “From this table, we observe that generative QA models (e.g., REAG, PALM) are consistently superior to extractive models (e.g., BiDAF) in answer quality.”
• The Large Language Model GreekLegalRoBERTa (2024, unknown): “Some outstanding examples of these models are OPT[34], BLOOM [28] and GPT3 [5], while the most recent and prominent are PaLM[4], Chinchilla[16] and Llama[31].”