Pre-trained Summarization Distillation

By Sam Shleifer, Alexander M. Rush. 2020

125 citations recorded by the source corpus.

Recent state-of-the-art approaches to summarization utilize large pre-trained Transformer models. Distilling these models to smaller student models has become critically important for practical use; however there are many different distillation methods proposed by the NLP literature. Recent work on distilling BERT for classification and regression tasks shows strong performance using direct knowledge distillation. Alternatively, machine translation practitioners distill using pseudo-labeling, where a small model is trained on the translations of a larger model. A third, simpler approach is to 'shrink and fine-tune' (SFT), which avoids any explicit distillation by copying parameters to a smaller student model and then fine-tuning. We compare these three approaches for distillation of Pegasus and BART, the current and former state of the art, pre-trained summarization models, and find that SFT outperforms knowledge distillation and pseudo-labeling on the CNN/DailyMail dataset, but under-performs pseudo-labeling on the more abstractive XSUM dataset. PyTorch Code and checkpoints of different sizes are available through Hugging Face transformers here this http URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Efficient Large Scale Language Modeling with Mixtures of Experts (2021, positive): “Building on recent work in this area (Shleifer and Rush, 2020; Sanh et al., 2020; Fedus et al., 2021), we train small dense “student” models to mimic the behavior of larger “teacher” models, which may be either large dense or large sparse (MoE) models.”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “Finally, we note that there are many optimizations, beyond seeking more efficient attention mechanisms, that are used to improve Transformer performance from fusing operations and sharing key-value-query matrices to general pruning, quantization, and distillation; see, for example, (Kim and Awadalla”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “, LSTM), the student models can be efficiently fine-tuned for better generation performance [26, 167].”
Block Pruning For Faster Transformers (2021, methodology): “Mo-bileBERT (Sun et al., 2020) is the result of a large architecture search. dBART (Shleifer and Rush, 2020) is obtained by arbitrarily copying equally spaced layers of a large model to a smaller one.”