Pre-trained models for natural language processing: A survey

By Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, Xuanjing Huang. 2020

1,716 citations recorded by the source corpus.

Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next, we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhiyuan Liu
3. Mohit Bansal
4. Maosong Sun
5. Minlie Huang
6. Graham Neubig
7. Xipeng Qiu
8. Percy Liang

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jianfeng Gao
3. Luc Van Gool
4. Yi Yang
5. Xiaogang Wang
6. Andrew Zisserman
7. Trevor Darrell
8. Cordelia Schmid

In Large Language Models

1. Jie Tang
2. Zhiyuan Liu
3. Minlie Huang
4. Weizhu Chen
5. Yu Cheng
6. Xiao Liu
7. Ming Ding
8. Jianfeng Gao

Citations in the corpus

• AI and the Everything in the Whole Wide World Benchmark (2021, background): “…neither systematically map out a range of specific linguistic skills required for understanding (lacking in particular any exploration of pragmatics [Qiu et al., 2020], or the proper handling of negation [Ettinger, 2020]) nor present a truly varied range of ways to deploy linguistic knowledge in…”
• GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “In this section, we review related work to GLM-130B on topics of pre-training, transferring, and inference of pre-trained LLMs (Qiu et al., 2020; Bommasani et al., 2021).”
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, neutral): “In natural language processing, self-supervised language modeling [7, 16, 31, 34, 44] is adopted for pre-training on large-scale unlabeled corpora [54].”
• Pre-Trained Models: Past, Present and Future (2021, background): “The natural language processing (NLP) community was also aware of the potential of PTMs and started to develop PTMs for NLP tasks (Qiu et al., 2020).”
• Towards a Unified View of Parameter-Efficient Transfer Learning (2021, background): “Transfer learning from pre-trained language models (PLMs) is now the prevalent paradigm in natural language processing, yielding strong performance on many tasks (Peters et al., 2018; Devlin et al., 2019; Qiu et al., 2020).”
• PTR: Prompt Tuning with Rules for Text Classification (2021, background): “It is now a consensus of the NLP community to fine-tune PLMs for specific tasks instead of learning models from scratch (Qiu et al., 2020).”
• A Primer on Pretrained Multilingual Language Models (2021, neutral): “Our survey is thus different from existing surveys on cross-lingual word embedding models (Ruder et al., 2019), multilingual NMT models (Dabre et al., 2020) and pre-trained language models (Qiu et al., 2020; Kalyan et al., 2021) which do not focus on pretrained multi-lingual language models.”
• A Survey on Text Classification: From Traditional to Deep Learning (2020, background): “Pre-trained language models [198] effectively learn global semantic representation and significantly boost NLP tasks, including text classification.”
• Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach (2023, neutral): “In this part, we propose a unified instruction format containing three key aspects for recommendation, and then instantiate it for different interaction scenarios.”
• A Unified Generative Framework for Various NER Subtasks (2021, background): “Recently, the amazing performance gain achieved by PTMs (pre-trained models) (Qiu et al., 2020; Peters et al., 2018; Devlin et al., 2019; Dai et al., 2021; Yan et al., 2020) has attracted several attempts to pre-train a Seq2Seq model (Song et al., 2019; Lewis et al., 2020; Raffel et al., 2020).”
• Large Language Models: A Survey (2024, neutral): “Recent surveys on PLMs include [8], [27], [28].”
• DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates (2021, background): “This points to the fact that current SOTA LMs still fail to capture the subtleties of human dialogue when complex or nuanced interactions occur [18, 29].”
• Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “For example, Qiu et al. [191] summarized two generations of PLMs for the whole NLP domain and introduced various extensions and adaption approaches of PLMs. Kalyan et al. [106] gave a brief overview of the advances of self-supervised learning in PLMs. Han et al. [84] took a deep look into the…”
• Generative artificial intelligence: a systematic review and applications (2024, neutral): “The basic technique of pretraining transformers was introduced and used as a state-of-the-art technique by [106].”
• A Large-Scale Chinese Short-Text Conversation Dataset (2020, background): “The success of pre-training models has greatly advanced the research of natural language processing [21].”