ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

By Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shan Zhang, Jie Fu. arXiv.org, 2023

961 citations recorded by the source corpus.

Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies. The multi-agent-based approach enables a group of LLMs to synergize with an array of intelligent counterparts, harnessing their distinct capabilities and expertise to enhance efficiency and effectiveness in handling intricate tasks. In this paper, we construct a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models on open-ended questions and traditional natural language generation (NLG) tasks. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments. Our code is available at https://github.com/chanchimin/ChatEval.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/ec58a564fdda29e6a9a0a7bab5eeb4c290f716d7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Mixture-of-Agents Enhances Large Language Model Capabilities (2024, neutral): “Other similar works include (Chan et al., 2023).”
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, positive): “Building on this foundation, several architectures and techniques for multi-agent LLMs have emerged, including mixture-of-agent (Zhang et al., 2023b), role play (Wu et al., 2023), debating (Chan et al., 2023), and voting (Zhu et al., 2024b).”
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, background): “Among notable prior works, researchers have proposed multi-agent debating frameworks (Du et al., 2023; Liang et al., 2023; Chan et al., 2023; Xiong et al., 2023a) but such efforts are still largely limited to multiple instances of the same underlying language model.”
Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024, background): “For example, many applications [ Du et al. , 2023; Xiong et al. , 2023; Chan et al. , 2023 ] utilize multiple agents to debate a question to reach a consensus.”
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (2023, background): “Recent studies demonstrate that employing ChatGPT and GPT-4 for scoring and comparing LLM outputs can achieve high agreement with human evaluation while reducing costs (Liu et al., 2023; Li et al., 2023a; Chan et al., 2023; Xu et al., 2023).”
Cognitive Architectures for Language Agents (2023, background): “, 2023), debate (Chan et al., 2023; Liang et al., 2023b; Du et al., 2023), improved safety (Irving et al.”
A survey on large language model based autonomous agents (2023, positive): “They collaborate and interact according to specified instructions, forming a virtual team that facilitates each other’s work.”
Why Do Multi-Agent LLM Systems Fail? (2025, neutral): “While unit test generation aids verification in software engineering (Jain et al., 2024), creating a universal verification mechanism remains challenging.”
Multi-Agent Collaboration Mechanisms: A Survey of LLMs (2025, neutral): “[ 74] have conceptualized assemblies of agents as a group and focused on exploring the potential of their cooperation [18, 41, 105, 132] found social behaviors autonomously emerge within a group of agents.”
The rise and potential of large language model based agents: a survey (2025, positive): “An example is cooperation, which is achieved through volunteer behaviors [330], brainstorming discussions [327, 340], and project management [332].”
Evaluating Large Language Models at Evaluating Instruction Following (2023, methodology): “ChatEval (Chan et al., 2023) .”
ChatDev: Communicative Agents for Software Development (2023, background): “Furthermore, LLMs have demonstrated strong role-playing abilities (Li et al., 2023a; Park et al., 2023; Hua et al., 2023; Chan et al., 2023; Zhou et al., 2023b; Chen et al., 2023b,a; Cohen et al., 2023; Li et al., 2023b).”
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024, background): “Recently, employing language models as a judge has gained attention as a promising paradigm to mimic the depth and granularity that human evaluation offers (Zheng et al., 2023; Liu et al., 2023b; Li et al., 2023b; Chan et al., 2023; Ye et al., 2023).”
AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors in Agents (2023, positive): “To facilitate effective decision-making, previous research has investigated the impact of different communication structures among agents (Chan et al., 2023; Zhang et al., 2023b; Wu et al., 2023).”
Improving Factuality and Reasoning in Language Models through Multiagent Debate (2023, positive): “In contrast, our work and work in (Liang et al., 2023; Chan et al., 2023) focus on using debate at inference time to improve the performance of language models.”
Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, positive): “ChatEval (Chan et al., 2024) present a multi-agent framework for text quality evaluation, employing LLMs as diverse role-playing agents, in which key roles include the public, critics, journalists, philosophers, and scientists, each contributing unique perspectives.”
STMA: A Spatio-Temporal Memory Agent for Long-Horizon Embodied Task Planning (2025, unknown): “The introduction of LLMs has significantly enhanced the reasoning and task-handling capabilities of agents [33–35].”