A Survey of Large Language Models
By Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong and 12 more. 2023
4,706 citations recorded by the source corpus.
The rapid evolution of large language models (LLMs) has driven a transformative shift in artificial intelligence (AI), reshaping both research paradigms and practical applications. Distinguished from their predecessors by unprecedented scale and advanced capabilities, LLMs necessitate new frameworks for understanding their development, behavior, and societal impact. This survey systematically reviews recent advancements in LLM techniques across four key dimensions: (1) pre-training methodologies, which establish core model capabilities through large-scale self-supervised training, architectural innovations, and data curation strategies; (2) post-training techniques, including supervised fine-tuning and reinforcement learning, which adapt foundational models to downstream tasks and enhance their alignment and safety; (3) utilization strategies, such as in-context learning, prompt engineering, and agentic reasoning, that optimize real-world deployment and enable effective interaction with external environments; and (4) evaluation methods, encompassing benchmarks for key ability dimensions such as core language capabilities, reasoning, and safety, which support comprehensive and reliable assessment of model performance. Additionally, we identify critical research issues, including those concerning theoretical foundations, efficient scaling, alignment, and agentic capability, and highlight the open challenges they present. By synthesizing state-of-the-art insights and emerging trends, this survey aims to provide a systematic and comprehensive framework for understanding the trajectory, current limitations, and future directions of LLM progress.
Subfields: Artificial Intelligence, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Luke S. Zettlemoyer
- Yoshua Bengio
- Dacheng Tao
- Percy Liang
- Quoc V. Le
- Graham Neubig
In Large Language Models
- Weizhu Chen
- Xiao Liu
- Kalpesh Krishna
- Aleksandra Piktus
- Yang Liu
- Alexander M. Rush
- Niklas Muennighoff
- Jean Kaddour
Citations in the corpus
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024, background): “Recent advancements in Large Language Models (LLMs) have significantly enhanced their capabilities across various natural language processing tasks, especially in systems that demand both factual accuracy and sophisticated reasoning for complex queries (Zhao et al., 2023; Krishna et al., 2024).”
- Editing Large Language Models: Problems, Methods, and Opportunities (2023, neutral): “Large language models (LLMs) have demonstrated a remarkable capacity for understanding and generating human-like text (Brown et al., 2020; OpenAI, 2023; Anil et al., 2023; Touvron et al., 2023; Qiao et al., 2022; Zhao et al., 2023).”
- Large Language Models are Zero-Shot Rankers for Recommender Systems (2023, background): “Existing studies show that LLMs can follow natural language instructions to solve diverse tasks in a zero-shot or few-shot setting [32, 37] Parsing the output of LLMs.”
- Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, neutral): “For example, to account for the impact of randomness originating from human decision-making, we can leverage discrete choice models for simulating pedestrian walking behaviors (Antonini et al., 2006).”
- Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, background): “Unlike previous ML algorithms, these large-scale models with a huge amount of parameters have shown versatile comprehension and reasoning capabilities in various fields such as health care [7], law [8], finance [9], education, and so on [10].”
- A Survey on Model Compression for Large Language Models (2023, neutral): “At the same time, Recent research [Wei et al. , 2022a; Schaeffer et al. , 2023; Zhao et al. , 2023] underscores that with the emphasis on augmenting model size, LLMs like GPT-3 (175B parameters) and PaLM (540B parameters) showcase unique behaviors when compared to smaller models like BERT (330M…”
- OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (2023, methodology): “Recently, there have been notable advancements in Large Language Models (LLMs), such as GPT-4 (OpenAI, 2023) and Chinchilla (Hoffmann et al., 2022), demonstrating impressive performance in various downstream natural language processing (NLP) tasks (Zhao et al., 2023).”
- LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked (2023, methodology): “Our evaluation of LLM S ELF D EFENSE focuses on two prominent language models: GPT 3.5 and Llama 2 7B. GPT 3.5 is one of the most popular LLMs, while Llama 2 is one of the most prominent open-source LLMs [32].”
- Adapted large language models can outperform medical experts in clinical text summarization (2023, positive): “This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.”
- A survey on large language model based autonomous agents (2023, neutral): “Ac-tually, simulating the human negative aspects can be even more important, since an important goal of simulation is to discover and solve problems, and without negative aspects means no problem to be solved.”
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “More details of LLM evaluation and Multi-model LLM evaluation can be found in recent surveys [10,67].”
- Recommender Systems in the Era of Large Language Models (LLMs) (2023, neutral): “Considering the suitable prompting strategies for adapting LLMs to various downstream tasks with complex reasoning, Zhao et al. [16] discuss the combination of ICL and CoT prompting under two major settings: few-shot CoT and zero-shot CoT, as illustrated below.”
- Towards Making the Most of ChatGPT for Machine Translation (2023, background): “Large language models (LLMs) usually refer to language models with hundreds of billions of parameters, which are trained on massive text data (Zhao et al., 2023).”
- Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “Pre-trained language models (PLMs) have demonstrated impressive capabilities in solving various NLP tasks [130, 246, 291, 338].”