LLM Evaluators Recognize and Favor Their Own Generations

By Arjun Panickssery, Samuel R. Bowman, Shi Feng. Neural Information Processing Systems, 2024

594 citations recorded by the source corpus.

Self-evaluation using large language models (LLMs) has proven valuable not only in benchmarking but also methods like reward modeling, constitutional AI, and self-refinement. But new biases are introduced due to the same LLM acting as both the evaluator and the evaluatee. One such bias is self-preference, where an LLM evaluator scores its own outputs higher than others' while human annotators consider them of equal quality. But do LLMs actually recognize their own outputs when they give those texts higher scores, or is it just a coincidence? In this paper, we investigate if self-recognition capability contributes to self-preference. We discover that, out of the box, LLMs such as GPT-4 and Llama 2 have non-trivial accuracy at distinguishing themselves from other LLMs and humans. By fine-tuning LLMs, we discover a linear correlation between self-recognition capability and the strength of self-preference bias; using controlled experiments, we show that the causal explanation resists straightforward confounders. We discuss how self-recognition can interfere with unbiased evaluations and AI safety more generally.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/5c7f465d162aade4a4c0eefb02fd7aadeebdaf58

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Prateek Yadav

Citations in the corpus

From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, unknown): “Lee et al. (2024) devise Judge augmented Supervised Fine-Tuning (JSFT) to train a single model to act as both a policy and a judge.”
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (2024, background): “…2, higher than its actual position 4, which is in line with previous works that have also observed GPT-4’s preference for its own generations (Zheng et al., 2024; Panickssery et al., 2024) Figure 4: Accuracy changes of different evaluation judges as compared to human judgements on Bam-boogle.”
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, background): “As previous studies have shown that LLMs favor their own generations [70, 47], such phenomena can make the model evaluation unfair.”