Knowledge Inheritance for Pre-trained Language Models

By Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun and 1 more. 2021

65 citations recorded by the source corpus.

Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger PLMs. However, it requires tremendous computational resources to train a large-scale PLM, which may be practically unaffordable. In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring that many well-trained PLMs are available. To this end, we explore the question how could existing PLMs benefit training large-scale PLMs in future. Specifically, we introduce a pre-training framework named “knowledge inheritance” (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs. Experimental results demonstrate the superiority of KI in training efficiency. We also conduct empirical analyses to explore the effects of teacher PLMs’ pre-training settings, including model architecture, pre-training data, etc. Finally, we show that KI could be applied to domain adaptation and knowledge transfer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Minlie Huang
4. Xiaoyan Zhu
5. Xu Han
6. Qun Liu
7. Jie Zhou
8. Yankai Lin

In Computer Vision and Pattern Recognition

1. Maosong Sun
2. Zhiyuan Liu
3. Qun Liu
4. Yang Liu
5. Xin Jiang
6. Xu Han
7. Minlie Huang
8. Peng Li

Citations in the corpus

• ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, background): “Soft prompts have been demonstrated as excellent task indicators (Qin et al., 2021b) and have non-trivial transferability among tasks (Su et al., 2021).”
• CPM-2: Large-scale Cost-effective Pre-trained Language Models (2021, methodology): “In (Qin et al., 2021), we introduce the overall framework of knowledge inheritance, indicating the effect of teacher PLMs’ settings, including pre-training methods, model architectures, training data, etc.”
• On Transferability of Prompt Tuning for Natural Language Understanding (2021, methodology): “…and measuring the essential factors of prompts, which needs a deeper understanding on the mechanisms of PLMs. Previous works (Aghajanyan et al., 2021; Qin et al., 2021b) show that various NLP tasks can be reparameter-ized into similar low-dimensional subspaces conditioning on PLMs, which may help…”
• bert2BERT: Towards Reusable Pretrained Language Models (2021, background): “Another line of work proposes to “back distil” the knowledge of the small models into large models, which is termed as knowledge inheritance (Qin et al., 2021).”
• Transformers in Time-Series Analysis: A Tutorial (2022, methodology): “These techniques include layer-wise adaptive large batch optimization [103], distributed training [37], knowledge inheritance [68], progressive training [46], and mapping parameters of smaller”