Pre-Trained Models: Past, Present and Future

By Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang and 12 more. 2021

1,095 citations recorded by the source corpus.

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can effectively capture knowledge from massive labeled and unlabeled data. By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks, which has been extensively demonstrated via experimental verification and empirical analysis. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch. In this paper, we take a deep look into the history of pre-training, especially its special relation with transfer learning and self-supervised learning, to reveal the crucial position of PTMs in the AI development spectrum. Further, we comprehensively review the latest breakthroughs of PTMs. These breakthroughs are driven by the surge of computational power and the increasing availability of data, towards four important directions: designing effective architectures, utilizing rich contexts, improving computational efficiency, and conducting interpretation and theoretical analysis. Finally, we discuss a series of open problems and research directions of PTMs, and hope our view can inspire and advance the future study of PTMs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Representation & Self-Supervised Learning

Citations in the corpus

GPT Understands, Too (2021, methodology): “Self-supervised (Liu et al., 2020) pre-trained language models (Han et al., 2021) including GPT (Rad-ford et al., 2019), BERT (Devlin et al., 2018), XL-Net (Yang et al., 2019), RoBERTa (Liu et al., 2019) have been observed to learn not only contextualized text representations but also linguistic…”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, background): “Recent years have witnessed interest in large-scale crossmodal pre-training [83].”
A systematic evaluation of large language models of code (2022, background): “Large-scale pretraining of LMs has had an astounding impact on natural language processing in recent years (Han et al., 2021).”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “pressive results in downstream tasks [38], [150].”
PPT: Pre-trained Prompt Tuning for Few-shot Learning (2021, methodology): “We follow Zhang et al. (2021) and Gao et al. (2021) to use the original validation set as the test set D test , which means Hybrid Prompt Tuning In hybrid prompt tuning, both soft and hard prompts are used (Liu et al., 2021; Han et al., 2021b).”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “Algorithms for learning language models (notably transformer LMs) show an unprecedented performance on extremely large models with hundreds of billions of parameters trained on extremely large datasets [7, 37, 20, 100, 45].”
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm (2021, background): “The critical idea of pre-training is to first extract general knowledge implicitly from the massive amount of data and then transfer the knowledge to versatile downstream tasks (Han et al., 2021).”
Tool Learning with Foundation Models (2023, neutral): “In recent years, the field of natural language processing (NLP) has undergone a paradigm shift, marked by the advent of pre-trained language models (PLMs) (Devlin et al., 2019; Bommasani et al., 2021; Han et al., 2021).”
AI literacy and its implications for prompt engineering strategies (2024, background): “Other research has concentrated on the prompt design for classification tasks and literature queries (Han et al., 2021).”
Prompt-Learning for Fine-Grained Entity Typing (2021, background): “Despite the success of ﬁne-tuning PLMs, the huge objective form gap between pre-training and ﬁne-tuning still hinders the full use of per-trained knowledge for downstream tasks (Liu et al., 2021b; Han et al., 2021b; Hu et al., 2021).”
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, background): “In recent years, large language models (LLMs) (Brown et al., 2020; OpenAI, 2023a; Touvron et al., 2023) have exhibited exceptional performance across a range of natural language processing (NLP) tasks (Qiu et al., 2020; Han et al., 2021).”
Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach (2023, neutral): “In this part, we propose a unified instruction format containing three key aspects for recommendation, and then instantiate it for different interaction scenarios.”
Federated Learning from Pre-Trained Models: A Contrastive Learning Approach (2022, background): “Due to the huge number of parameters and the broad data available for training, pre-trained foundation models can better capture knowledge for downstream tasks and lead to green AI [52, 53].”
A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation (2023, methodology): “Pre-trained models such as BERT and Chat-GPT have demonstrated remarkable capabilities in understanding complex linguistic patterns and semantics [346, 347].”
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (2021, background): “Recent years have witnessed the prominence of Pretrained Language Models (PLMs) (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Raffel et al., 2020; Xu et al., 2021) due to their superior performance on a wide range of languagerelated downstream tasks such as text classification (Ko”