LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
By Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, Yiqun Liu. arXiv.org, 2024
537 citations recorded by the source corpus.
The rapid advancement of Large Language Models (LLMs) has driven their expanding application across various fields. One of the most promising applications is their role as evaluators based on natural language responses, referred to as ''LLMs-as-judges''. This framework has attracted growing attention from both academia and industry due to their excellent effectiveness, ability to generalize across tasks, and interpretability in the form of natural language. This paper presents a comprehensive survey of the LLMs-as-judges paradigm from five key perspectives: Functionality, Methodology, Applications, Meta-evaluation, and Limitations. We begin by providing a systematic definition of LLMs-as-Judges and introduce their functionality (Why use LLM judges?). Then we address methodology to construct an evaluation system with LLMs (How to use LLM judges?). Additionally, we investigate the potential domains for their application (Where to use LLM judges?) and discuss methods for evaluating them in various contexts (How to evaluate LLM judges?). Finally, we provide a detailed analysis of the limitations of LLM judges and discuss potential future directions. Through a structured and comprehensive analysis, we aim aims to provide insights on the development and application of LLMs-as-judges in both research and practice. We will continue to maintain the relevant resource list at https://github.com/CSHaitao/Awesome-LLMs-as-Judges.
Subfields: Artificial Intelligence, Law, Political Science and International Relations
View Paper: https://www.semanticscholar.org/paper/62f441d5078bf77927c370364367c20f4e0010e6
Citations in the corpus
- Measuring Semantic Progress in Multi-turn Dialogue via Information Gain (2026, unknown): “We focus on semantic progress as one evaluation dimension; factuality, safety, coherence, and style are each studied by extensive evaluation literature [11, 12] and are complementary to the signal proposed here.”
- Efficient Context-Aware Corpus-Grounded Query and Prompt Suggestion constrained by Creative Knowledge Graph (2026, unknown): “Recent work has investigated the use of LLMs as automated evaluators—commonly referred to as the “LLM-as-a-judge” paradigm—for scoring generation quality across multiple dimensions [3, 15].”
- SurveyGen-I: Consistent Scientific Survey Generation with Evolving Plans and Memory-Guided Writing (2025, unknown): “Each aspect is scored by LLM-as-Judge (Li et al., 2024a) (specifically, rated by GPT4o-mini), with explanation-based prompts to reduce variance.”
- InteractiveSurvey: An LLM-based Personalized and Interactive Survey Paper Generation System (2025, unknown): “Evaluation Metrics Following existing survey generation systems (Wang et al., 2024b; Liang et al., 2025), we employ LLMs as judges (Li et al., 2024a) to assess the survey papers in Coverage : the extent to which the survey encapsulates all aspects of the topic; Structure : the logical organization…”
- JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols (2026, unknown): “Specifically, LLM judgments can take different forms, which can be broadly categorized into three evaluation protocols: pointwise scoring, pairwise comparison, and listwise ranking, each providing a distinct form of supervision [22,45].”