A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

By Ce Zhou, Qian Li, Chen Li, Guangjing Wang, Kaichao Zhang, Cheng Ji, Qi Yan, Lifang He, Hao Peng, Jianxin Li and 15 more. arXiv.org, 2023

736 citations recorded by the source corpus.

Pretrained Foundation Models (PFMs) are regarded as the foundation for various downstream tasks with different data modalities. A PFM (e.g., BERT, ChatGPT, and GPT-4) is trained on large-scale data which provides a reasonable parameter initialization for a wide range of downstream applications. BERT learns bidirectional encoder representations from Transformers, which are trained on large datasets as contextual language models. Similarly, the generative pretrained transformer (GPT) method employs Transformers as the feature extractor and is trained using an autoregressive paradigm on large datasets. Recently, ChatGPT shows promising success on large language models, which applies an autoregressive language model with zero shot or few shot prompting. The remarkable achievements of PFM have brought significant breakthroughs to various fields of AI. Numerous studies have proposed different methods, raising the demand for an updated survey. This study provides a comprehensive review of recent research advancements, challenges, and opportunities for PFMs in text, image, graph, as well as other data modalities. The review covers the basic components and existing pretraining methods used in natural language processing, computer vision, and graph learning. Additionally, it explores advanced PFMs used for different data modalities and unified PFMs that consider data quality and quantity. The review also discusses research related to the fundamentals of PFMs, such as model efficiency and compression, security, and privacy. Finally, the study provides key implications, future research directions, challenges, and open problems in the field of PFMs. Overall, this survey aims to shed light on the research of the PFMs on scalability, security, logical reasoning ability, cross-domain learning ability, and the user-friendly interactive ability for artificial general intelligence.

Subfields: Artificial Intelligence, Management Information Systems, Political Science and International Relations

View Paper: https://www.semanticscholar.org/paper/3599a236f285af48782fc30b1341d13ec7320735

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Philip S. Yu
2. Lichao Sun
3. Zhiling Yan
4. Siyu Li
5. Yihan Cao
6. Yutong Dai
7. Yu Su
8. Huan Sun

Citations in the corpus

• Segment Anything Model for medical image analysis: an experimental study (2023, background): “These difficulties could be significantly mitigated with the advent of foundation models [39] and zero-shot learning [6].”
• A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, methodology): “Within the context of the AIGC framework, a fundamental component of each proposed method involves utilizing large pretrained foundation models (PFMs) [212].”
• SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (2023, background): “Recently, Foundation Model emerges as a pre-trained generic model that excels in a wide range of downstream tasks [82, 88].”
• A Categorical Archive of ChatGPT Failures (2023, background): “It is commonly held that Large Language Models (LLMs) [9, 16, 54, 67] will have a far-reaching impact and be employed as aides by numerous professionals.”
• A Comprehensive Overview of Large Language Models (2023, neutral): “1: The trends in the number of LLM models introduced over the years.”
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (2023, background): “Conversational large language models (LLMs) [1], such as ChatGPT [2], have generated immense interest in a range of domains for tasks ranging from answering questions on medical licensing exams [3] to generating code snippets.”
• Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “Related Reviews and Differences: In the literature, few recent works have reviewed large language models (LLMs) in natural language processing [2], [20], [21], [22], [23].”
• Large Language Models: A Survey (2024, neutral): “Recent surveys on PLMs include [8], [27], [28].”
• Mind2Web: Towards a Generalist Agent for the Web (2023, background): “These models, often encompassing billions of parameters, are pre-trained on massive corpora of text data [3, 44, 45], enabling them to capture intricate linguistic patterns, nuances, and relationships, resulting in unprecedented performance on a wide array of NLP tasks.”
• A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, background): “General Review: The impressive success of pretrained language models (Radford et al., 2019; Devlin et al., 2018; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020; Liu et al., 2023c; Zhou et al., 2023) has led to the development of several conversational language models, including, Meena (Ad”