Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey

By Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, Dan Roth. ACM Computing Surveys, 2021

1,575 citations recorded by the source corpus.

Large, pre-trained language models (PLMs) such as BERT and GPT have drastically changed the Natural Language Processing (NLP) field. For numerous NLP tasks, approaches leveraging PLMs have achieved state-of-the-art performance. The key idea is to learn a generic, latent representation of language from a generic task once, then share it across disparate NLP tasks. Language modeling serves as the generic task, one with abundant self-supervised text available for extensive training. This article presents the key fundamental concepts of PLM architectures and a comprehensive view of the shift to PLM-driven NLP techniques. It surveys work applying the pre-training then fine-tuning, prompting, and text generation approaches. In addition, it discusses PLM limitations and suggested directions for future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c23d9d44e8bc68408cea9f305d1f24d915bc0d0d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

OpenAGI: When LLM Meets Domain Experts (2023, background): “With the advancement of highly parallelizable transformer architectures, pre-trained language models (PLMs) have demonstrated remarkable capabilities in comprehending, generating, and manipulating natural language [23, 18].”
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023, background): “Since fine-tuning and prompting of Large Language Models (LLMs) became commonplace (Min et al., 2021) it has been increasingly difficult to enforce those strict protocols.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “Other surveys [17,18] include discussions on technological aspects but primarily focus on general LLM developments and evaluations, offering limited insights into their adaptation and application in healthcare settings.”
Knowledge Inheritance for Pre-trained Language Models (2021, neutral): “Recently, it has become a consensus in the NLP community to use pre-trained language models (PLMs) as the backbone for various downstream tasks (Han et al., 2021; Min et al., 2021).”
Bias and Fairness in Large Language Models: A Survey (2023, background): “This "pre-train, then fine-tune" paradigm enables the training of one foundation model that can be adapted to a range of applications (Bommasani et al., 2021; Min et al., 2023).”
Recommender Systems in the Era of Large Language Models (LLMs) (2023, neutral): “......”
Refusal in Language Models Is Mediated by a Single Direction (2024, neutral): “T HE P ILE consists of scraped webtext, and so we do not append any chat template when evaluating CE loss.”
A Comprehensive Overview of Large Language Models (2023, neutral): “1: The trends in the number of LLM models introduced over the years.”
Practical and ethical challenges of large language models in education: A systematic scoping review (2023, background): “…to selectively attend to the different parts of input texts, depending on the focus of the current tasks, allowing the model to learn complex patterns and relationships among textual contents, such as their semantic, contextual, and syntactic relationships (Min et al., 2021; Liu et al., 2023).”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, neutral): “25, 26, 27] or in-context few-shot learning methods [28, 29].”
The debate over understanding in AI’s large language models (2022, background): “Such models can be used to generate natural language, be fine-tuned for specific language tasks (3), or be further trained to better match “user intent” (4).”
Multi-Agent Collaboration Mechanisms: A Survey of LLMs (2025, neutral): “Moreover, the geopolitical and ethical implications of LLM development have become the limelight for policymakers, especially concerning the computational power required and potential misuse by malicious actors [76, 86].”