Leveraging Pre-trained Checkpoints for Sequence Generation Tasks

By Sascha Rothe, Aliaksei Severyn. 2019

471 citations recorded by the source corpus.

Abstract Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing. By warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language Understanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We developed a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT, GPT-2, and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentence Fusion.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Big Bird: Transformers for Longer Sequences (2020, methodology): “As has been pointed out recently [76, 108], pretraining helps in generative tasks, we warm start from our general purpose MLM pretraining on base-sized models as well as utilizing state-of-the-art summarization speciﬁc pretraining from Pegasus [108] on large-sized models.”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, background): “…has led to a dramatic improvement in performance when fine-tuned for both natural language understanding and text generation tasks (Radford et al., 2018a; Devlin et al., 2019; Rothe et al., 2019; Yang et al., 2019; Joshi et al., 2019; Song et al., 2019; Dong et al., 2019; Lewis et al., 2019).”
Learning to summarize from human feedback (2020, background): “A human judges which is a better summary of the post.”
On Faithfulness and Factuality in Abstractive Summarization (2020, methodology): “All weights in TRANS2S are randomly initialized, but in BERTS2S, both encoder and decoder are initialized with the BERT-Base checkpoints (Devlin et al., 2019), with parameter sharing between the encoder and decoder, following Rothe et al. (2020).”
DART: Open-Domain Structured Data Record to Text Generation (2020, neutral): “, 2019), BERTto-BERT (Rothe et al., 2020), DialoGPT (Zhang et al.”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, methodology): “(2), inspired by (Rothe et al., 2020; Zhou et al., 2020) that uses BERT as a decoder, one can re-use language model heads as prediction heads for generation.”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, methodology): “Both the BERT-to-BERT models perform the best, followed by the pointer generator model.”
Text Summarization with Pretrained Encoders (2019, background): “Very recently, there have been attempts to apply pretrained models to various generation problems (Edunov et al., 2019; Rothe et al., 2019).”
ETC: Encoding Long and Structured Inputs in Transformers (2020, methodology): “Speciﬁcally, we initialized from the RoBERTa checkpoints reported in the work of Rothe et al. (Rothe et al., 2020).”
Text-to-Text Pre-Training for Data-to-Text Tasks (2020, methodology): “• BERT-to-BERT (Rothe et al., 2019) - A transformer based encoder-decoder model, where both the encoder and decoder are initialized with BERT.”
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2020, methodology): “2, summaries generated from BERTS2S contain signiﬁcantly fewer hallucinations than other model outputs.”
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021, methodology): “…Transformer (Vaswani et al., 2017) encoder-decoder model fine-tuned on the XSum dataset BERTS2S Transformer encoder-decoder model with parameter sharing (Rothe et al., 2020) where both encoder and decoder are initialized with the BERT-Base checkpoints (Devlin et al., 2019) and fine-tuned on XSum”
Neural CRF Model for Sentence Alignment in Text Simplification (2020, positive): “The encoder is initialized with the same checkpoint and the decoder is randomly initialized (Rothe et al., 2020).”
Planning with Learned Entity Prompts for Abstractive Summarization (2021, background): “Recent encoder-decoder models for text generation (Bahdanau et al., 2015; Sutskever et al., 2014; Vaswani et al., 2017; Rothe et al., 2020; Lewis et al., 2020; Raffel et al., 2019; Zhang et al., 2020) tend to perform text generation in an end-to-end setting, which means that most approaches do not…”
Multi-stage Pretraining for Abstractive Summarization (2019, background): “Pretraining has been popular for image classification (He et al., 2018), language understanding, see most recently (Devlin et al., 2018), and language generation (Ramachandran et al., 2017; Edunov et al., 2019; Rothe et al., 2019) tasks.”
Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data (2020, unknown): “Deep neural network-based (DNN) models have demonstrated remarkable performance on a multitude of text-to-text (Bahdanau et al., 2015; Rothe et al., 2019; Narayan et al., 2018; Rush et al., 2015, inter alia) as well as data-to-text generation tasks (Wiseman et al., 2017; Puduppully et al., 2019,…”
Plan-then-Generate: Controlled Data-to-Text Generation via Planning (2021, methodology): “Because BERT-to-BERT and T5-3B do not take the content plan as input, thus we only measure the accuracy score for the results generated by our model and the reference sentence.”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, background): “, machine translation [50], question answering [45], text summarization [27], etc.”