Large Language Models: A Survey

By Shervin Minaee, Tomas Mikolov, Narjes Nikzad, M. Chenaghlu, Richard Socher, Xavier Amatriain, Jianfeng Gao. arXiv.org, 2024

1,030 citations recorded by the source corpus.

Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/a1f76db91c0debcf93ae9889736bce8470902113

Citations in the corpus

• A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, neutral): “…al. [107] also conduct LLM surveys from the holistic view, which, however, provide different taxonomies and understandings on LLMs. Minaee et al. [108] further enrich this landscape by reviewing prominent LLM families and discussing their characteristics, while Huang et al. [53] specifically…”
• A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (2025, neutral): “Despitetheexponentialgrowthinbenchmarksformodelreasoning abilities 31 , the evaluation of LLMs on clinical tasks has typically been carried out via “ question-answering ” (QA) benchmarks 5,8,32 .”
• This is Going to Sound Crazy, But What If We Used Large Language Models to Boost Automatic Database Tuning Algorithms By Leveraging Prior History? We Will Find Better Configurations More Quickly Than Retraining From Scratch! (2025, unknown): “However, providing curated information to the LLM remains unsolved [26, 55, 90].”
• Measuring the Feasibility of a Question and Answering System for the Sarawak Gazette Using Chatbot Technology (2025, unknown): “Med-PaLM itself was improved into Med-PaLM 2 via fine-tuning and ensemble prompting, setting a new state-of-the-art on the MedQA dataset with a score of 86.5% (Minaee et al, 2024).”
• Human or Machine? A Survey on Machine-Generated Text Detection (2026, unknown): “As shown in Figure 2, most of the corpora were developed using a limited number of LLMs and mainly used models from the GPT family, so there is a need to use open-source models like CODEX, Alpaca, Koala, FLAN, Zephyr, DeepSeek, MT5 [125], etc.”