Large Language Models are not Fair Evaluators

By Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, Zhifang Sui. 2023

1,116 citations recorded by the source corpus.

In this paper, we uncover a systematic bias in the evaluation paradigm of adopting large language models~(LLMs), e.g., GPT-4, as a referee to score and compare the quality of responses generated by candidate models. We find that the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other, e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator. To address this issue, we propose a calibration framework with three simple yet effective strategies: 1) Multiple Evidence Calibration, which requires the evaluator model to generate multiple evaluation evidence before assigning ratings; 2) Balanced Position Calibration, which aggregates results across various orders to determine the final score; 3) Human-in-the-Loop Calibration, which introduces a balanced position diversity entropy to measure the difficulty of each example and seeks human assistance when needed. We also manually annotate the"win/tie/lose"outcomes of responses from ChatGPT and Vicuna-13B in the Vicuna Benchmark's question prompt, and extensive experiments demonstrate that our approach successfully mitigates evaluation bias, resulting in closer alignment with human judgments. We release our code and human annotation at \url{https://github.com/i-Eval/FairEval} to facilitate future research.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “This approach has been tried in our earlier blog post [8] and other concurrent or follow-up work [5, 28, 14, 12, 51, 18, 32, 39, 7, 42].”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, neutral): “The order in which candidates are shown to an LLM can bias which candidate it prefers (Pezeshkpour and Hruschka, 2023; Wang et al., 2023).”
A Survey on Evaluation of Large Language Models (2023, background): “Moreover, in the assessment of GPT4 alignment, (Wang et al., 2023e) discovered a systematic bias.”
Detecting and Preventing Hallucinations in Large Vision Language Models (2023, methodology): “However, using GPT-4 as a refree to score quality of responses is plagued with systematic bias such as senstitivity to positioning/ordering of responses in the context [26].”
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents (2023, background): “However, we remind readers that LLMs are known to have biases and problems for evaluation, including positional bias (Wang et al., 2023), factual inconsistency (Luo et al., 2023), favoring native speakers (Liang et al., 2023).”
Making Large Language Models Better Reasoners with Alignment (2023, positive): “To ensure the reliability of the ratings, following (Wang et al., 2023b), we require ChatGPT to present evaluation evidence before assigning a score, and simple 3 scores for each example.”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, background): “The order in which candidates are shown to an LLM can bias which candidate it prefers (Pezeshkpour & Hruschka, 2023; Wang et al., 2023).”
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2023, background): “This bias indicates that LLMs might not be a fair judge (Wang et al., 2023) when different LLMs are used for the agents.”
Preference Ranking Optimization for Human Alignment (2023, methodology): “To avoid unfairness in evaluation, we select two different reward models for training and evaluation, which we denote as RM train4 and RM eval5 , respectively.”
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2023, background): “Furthermore, taking GPT-4 as a referee suffers from a position bias [36], and simply changing the order of candidates could skew the evaluation result.”
VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model (2025, neutral): “Several literature [15, 33, 40, 41, 49, 51] also investigate this phenomenon in large language model research.”
ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, methodology): “To counter positional bias in GPT-4 ratings [Wang et al., 2023a], we collect two sets of ratings by alternating the order of test and baseline model responses.”
Aligning Large Language Models with Human: A Survey (2023, methodology): “The preliminary analysis in Wang et al. (2023d) also reveal that ShareGPT performs consistly well across a wide range of NLP tasks.”
Style Over Substance: Evaluation Biases for Large Language Models (2023, positive): “This intriguing finding is consistent with prior research (Wang et al., 2023), further underscoring the significance of this bias.”
RAGAs: Automated Evaluation of Retrieval Augmented Generation (2023, methodology): “However, care is needed with this approach, as the order in which the answers are presented can influence the result (Wang et al., 2023b).”