Benchmarking Cognitive Biases in Large Language Models as Evaluators

By Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, Dongyeop Kang. 2023

183 citations recorded by the source corpus.

Large Language Models are cognitively biased judges. Large Language Models (LLMs) have recently been shown to be effective as automatic evaluators with simple prompting and in-context learning. In this work, we assemble 15 LLMs of four different size ranges and evaluate their output responses by preference ranking from the other LLMs as evaluators, such as System Star is better than System Square. We then evaluate the quality of ranking outputs introducing the Cognitive Bias Benchmark for LLMs as Evaluators (CoBBLEr), a benchmark to measure six different cognitive biases in LLM evaluation outputs, such as the Egocentric bias where a model prefers to rank its own outputs highly in evaluation. We find that LLMs are biased text quality evaluators, exhibiting strong indications on our bias benchmark (average of 40% of comparisons across all models) within each of their evaluations that question their robustness as evaluators. Furthermore, we examine the correlation between human and machine preferences and calculate the average Rank-Biased Overlap (RBO) score to be 49.6%, indicating that machine preferences are misaligned with humans. According to our findings, LLMs may still be unable to be utilized for automatic annotation aligned with human preferences. Our project page is at: https://minnesotanlp.github.io/cobbler.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, unknown): “Length Bias , another prevalent challenge, skews evaluations by disproportionately favoring longer or shorter responses regardless of quality (Zheng et al., 2023; Koo et al., 2023a).”
Better & Faster Large Language Models via Multi-token Prediction (2024, methodology): “In order to avoid the need for human annotations of generation quality or language model judges—which comes with its own pitfalls, as pointed out by Koo et al. (2023)—we conduct evaluations on summarization and natural language mathematics benchmarks and compare pretrained models with training sets…”
Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024, methodology): “We investigate how various model names and anonymization strategies impact judgments, inspired by the observations of Koo et al. (2023).”
SeaLLMs - Large Language Models for Southeast Asia (2023, background): “This is despite the fact that GPT-4 evaluator favors ChatGPT more often than humans do [12].”
LLM Evaluators Recognize and Favor Their Own Generations (2024, unknown): “…et al., 2023; Zeng et al., 2023; Yuan et al., 2023; Fu et al., 2023; Li et al., 2024), self-refinement (Saunders et al., 2022; Madaan et al., 2023; Lee et al., 2023; Shridhar et al., 2023), and constitutional AI (Bai et al., 2022), LLMs are increasingly used to provide assessment, supervision,…”