Humans or LLMs as the Judge? A Study on Judgement Bias

By Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, Benyou Wang. Conference on Empirical Methods in Natural Language Processing, 2024

350 citations recorded by the source corpus.

Adopting human and large language models (LLM) as judges (*a.k.a* human- and LLM-as-a-judge) for evaluating the performance of LLMs has recently gained attention. Nonetheless, this approach concurrently introduces potential biases from human and LLMs, questioning the reliability of the evaluation results. In this paper, we propose a novel framework that is free from referencing groundtruth annotations for investigating **Misinformation Oversight Bias**, **Gender Bias**, **Authority Bias** and **Beauty Bias** on LLM and human judges. We curate a dataset referring to the revised Bloom’s Taxonomy and conduct thousands of evaluations. Results show that human and LLM judges are vulnerable to perturbations to various degrees, and that even the cutting-edge judges possess considerable biases. We further exploit these biases to conduct attacks on LLM judges. We hope that our work can notify the community of the bias and vulnerability of human- and LLM-as-a-judge, as well as the urgency of developing robust evaluation systems.

Subfields: Economics and Econometrics, Law, Strategy and Management

View Paper: https://www.semanticscholar.org/paper/a28071c63963cc59ba500cd00c140ac08eb5ccb0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Jiayi Ye
Yanbo Wang
Yue Huang
Qihui Zhang
Nuno Moniz
Tian Gao
Werner Geyer
Chao Huang

Citations in the corpus

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024, methodology): “We develop tests to evaluate this ability across various types of fallacies, contributing to fair and accurate judgments, as discussed in Chen et al. (2024c).”
HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025, neutral): “Length bias is a known issue of evaluations that rely on model-based graders (Ye et al., 2024; Chen et al., 2024).”
When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings (2024, unknown): “Finally, a large body of work compared cognitive biases of models to those of humans (Coda-Forno et al. (2024); Liu et al. (2024); Binz and Schulz (2023)).”