A Survey on Evaluation of Large Language Models
By Yu-Chu Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang and 6 more. 2023
3,582 citations recorded by the source corpus.
Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, education, natural and social sciences, agent applications, and other areas. Secondly, we answer the ‘where’ and ‘how’ questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing the performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey
Subfields: Artificial Intelligence, Health Informatics, Large Language Models, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Graham Neubig
- Mohit Bansal
- Luke S. Zettlemoyer
- Quoc V. Le
- Noah A. Smith
- Percy Liang
- Yoshua Bengio
In Health Informatics
- Mihai Surdeanu
- Jihan Yang
- Shahriar Golchin
- Raymond J. Mooney
- Nazneen Rajani
- Heng Ji
- Hinrich Schütze
- Pankaj Gupta
In Large Language Models
In Evaluation & Benchmarks
Citations in the corpus
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, background): “Other recent benchmarks, including MMLU [18], HELM [22], BigBench [32], Hel-laSwag [45], and the AI2 Reasoning Challenge (ARC) [12], have broadened the scope by assessing capabilities across language generation, knowledge understanding, and complex reasoning [9].”
- Aligning Large Language Models with Human: A Survey (2023, methodology): “We refer readers to Chang et al. (2023) for more comprehensive introduction of LLMs’ evaluation benchmarks.”
- Evaluation of Retrieval-Augmented Generation: A Survey (2024, neutral): “Searches in a dynamic database may lead to misleading results or contradicting the facts.”
- INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection (2024, methodology): “In the main experiments, we employ the ROUGE and sentence similarity as correctness measure, which are widely used for natural language generation evaluation (Chang et al., 2023; Kuhn et al., 2022; Huang et al., 2023).”
- Benchmarking Large Language Models in Retrieval-Augmented Generation (2023, background): “Evaluating LLMs has received significant attention due to their remarkable general capability (Chang et al. 2023).”
- A survey on large language model based autonomous agents (2023, background): “[15] shed light on evaluating LLMs, addressing what to evaluate, where to evaluate, and how to assess their performance in downstream tasks and societal impact.”
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, neutral): “…for a single task, and thus, hallucination studies on them are usually task-specific (Maynez et al., 2020; Wang and Sennrich, 2020; Xiao and Wang, 2021); however, current LLMs are expected to excel in multi-task, multi-lingual, and multi-domain settings (Bang et al., 2023; Chang et al., 2023).”
- Adapted large language models can outperform medical experts in clinical text summarization (2023, neutral): “In addition, beyond the scope of this manuscript, there is further potential to improve accuracy through incorporating checks by human operators and checks by other LLMs or using a model ensemble 47,48 .”
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models (2023, background): “…models such as the GPT-3/3.5 family (Brown et al. 2020) and GPT-4 (OpenAI 2023; Bubeck et al. 2023), and, needless to say, raises questions on the validity of evaluations and benchmarks conducted so far (Chang et al. 2023; Zhu et al. 2023; Bordt & von Luxburg 2023; Ray 2023; Penedo et al. 2023).”
- Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, methodology): “Manual evaluation for LLM alignment to human values generally offers a more holistic and precise assessment compared to automated evaluation [85].”
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023, background): “Hyperparameter Optimization The importance of hyperparameter optimization in machine learning [30, 31, 32, 33, 34, 35], particularly in the context of fine-tuning deep learning language models such as BERT [36] and GPT [37], cannot be ignored.”
- ChatGPT: The End of Online Exam Integrity? (2022, neutral): “…context, Liévin et al. [58] conclude that LLMs can effectively answer and reason about medical questions, while a recent survey Chang et al. [30] indicates that LLMs perform well in tasks like arithmetic reasoning and demonstrate marked competence in logical reasoning tasks too; though,…”
- A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, background): “Chang et al. [17] focuses on the assessment of LMMs.”