Style Over Substance: Evaluation Biases for Large Language Models

By Minghao Wu, Alham Fikri Aji. 2023

78 citations recorded by the source corpus.

As large language models (LLMs) continue to advance, accurately and comprehensively evaluating their performance becomes increasingly challenging. Ranking the relative performance of LLMs based on Elo ratings, according to human judgment, is gaining more popularity. However, the extent to which humans and LLMs are capable evaluators remains uncertain. This study investigates the behavior of crowd-sourced and expert annotators, as well as LLMs, when comparing outputs from different models. To achieve this, we curate a dataset of intentionally flawed machine-generated answers. Our findings reveal a concerning bias in the evaluation process, as answers with factual errors are rated more favorably than answers that are too short or contained grammatical errors. To address this issue, we propose independently evaluating machine-generated text across multiple dimensions, rather than merging all the evaluation aspects into a single score. We instantiate this idea with the Elo rating system, resulting in the Multi-Elo Rating System (MERS). Empirical results from our study reveal that this proposed approach significantly enhances the quality of LLM-based evaluations, particularly in terms of factual accuracy. However, there is no significant improvement in crowd-sourced-based evaluations, indicating the need for further investigation.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Jiayi Ye
2. Qun Liu
3. Xin Jiang
4. Yanbo Wang
5. Lifeng Shang
6. Yue Huang
7. Dongping Chen
8. Qihui Zhang

Citations in the corpus

• Humans or LLMs as the Judge? A Study on Judgement Bias (2024, methodology): “Recent works (Raina et al., 2024; Shi et al., 2024) propose optimization-based methods to hack LLM-as-a-judge.”
• Benchmarking Cognitive Biases in Large Language Models as Evaluators (2023, methodology): “Along this aspect, while our work is close to Wu and Aji (2023), who investigate biases related to fabricated factual and grammatical errors, our work is much more comprehensive in terms of the number of LLMs analyzed, the types of biases analyzed and the creation of an open benchmark.”
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024, background): “Additionally, Wu & Aji (2023) demonstrated that the limited size and scope of test data increase the risk of random interference, potentially obscuring the true extent of bias in LLM judgments.”
• Aligning Large Language Models with Human: A Survey (2023, background): “, 2023b), Multi-Elo (Wu and Aji, 2023) LLM-as-a-Judge (Zheng et al.”