Can Large Language Models Be an Alternative to Human Evaluations?

By Cheng-Han Chiang, Hung-yi Lee. 2023

1,037 citations recorded by the source corpus.

Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans. However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms.Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided.In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation.We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation.We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: open-ended story generation and adversarial attacks.We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs.We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer.We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “This approach has been tried in our earlier blog post [8] and other concurrent or follow-up work [5, 28, 14, 12, 51, 18, 32, 39, 7, 42].”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, methodology): “…2023a) have demonstrated their advanced capability to perform innovative applications, ranging from tool utilization, supplementing human evaluations, to stimulating human interactive behaviors (Bubeck et al., 2023; Schick et al., 2024; Chiang & Lee, 2023; Park et al., 2023; Jiao et al., 2023).”
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024, methodology): “The recent trend includes utilizing GPT-4 for approximating human judgment (Chiang & Lee, 2023), with notable instances being MT-Bench (Zheng et al., 2023b) and AlpacaEval (Li et al., 2023).”
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023, background): “In view of the impressive text understanding and instruction-following capabilities of recent LLMs, a body of literature (Liu et al., 2023b; Chiang & Lee, 2023; Gao et al., 2023; Shen et al., 2023) has adopted LLM as an evaluator to assess the quality of responses to open-ended questions or…”
Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, neutral): “According to Chiang & Lee (2023), GPT-4 can replace human evaluators in assessing the alignment capabilities of LLMs.”
Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “We also face issues like the need for extensive labeled data [347], significant computational resources [118, 248], and avoiding potential ethical problems [5, 40, 325].”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “Recently, the remarkable instruction-following ability of LLMs has underscored their potential for automatic evaluation (Chiang and Lee, 2023; Liu et al., 2023g; Wang et al., 2023d).”
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (2023, methodology): “We conduct an additional study of the quality of generated text using GPT4, given the fact that several prior studies Chiang & Lee (2023a); Liu et al. (2023) have shown the great potential of GPT-4 to serve as an alternative to human evaluation.”
Evaluating Large Language Models at Evaluating Instruction Following (2023, background): “…evaluation (Ouyang et al., 2022; Zhao et al., 2023; Wu et al., 2023a), many recent works take prompted LLMs as evaluators to compare model outputs (Chiang et al., 2023; Peng et al., 2023; Dubois et al., 2023; Zhou et al., 2023; Rafailov et al., 2023; Wang et al., 2023c; Xu et al., 2023; Song et…”
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, neutral): “Our MM-Vet is most related to the concurrent evaluation studies [26, 52, 44, 76] P r o p o r t i o n ( % ) 28.4 17.0 11.9 6.4 5.5 5.5 5.0 4.1 3.7 3.7 3.2 1.8 1.4 0.9 0.9 0.5”
Breaking Distortion-free Watermarks in Large Language Models (2025, unknown): “With these metrics as viable proxies for human preference [5, 34], we can confidently conclude that humans would be unlikely to notice any difference in quality, and the outputs are coherent and on-topic given the prompts.”