Evaluating Large Language Models at Evaluating Instruction Following

By Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, Danqi Chen. International Conference on Learning Representations, 2023

335 citations recorded by the source corpus.

As research in large language models (LLMs) continues to accelerate, LLM-based evaluation has emerged as a scalable and cost-effective alternative to human evaluations for comparing the ever increasing list of models. This paper investigates the efficacy of these ``LLM evaluators'', particularly in using them to assess instruction following, a metric that gauges how closely generated text adheres to the given instruction. We introduce a challenging meta-evaluation benchmark, LLMBar, designed to test the ability of an LLM evaluator in discerning instruction-following outputs. The authors manually curated 419 pairs of outputs, one adhering to instructions while the other diverging, yet may possess deceptive qualities that mislead an LLM evaluator, e.g., a more engaging tone. Contrary to existing meta-evaluation, we discover that different evaluators (i.e., combinations of LLMs and prompts) exhibit distinct performance on LLMBar and even the highest-scoring ones have substantial room for improvement. We also present a novel suite of prompting strategies that further close the gap between LLM and human evaluators. With LLMBar, we hope to offer more insight into LLM evaluators and foster future research in developing better instruction-following models.

View Paper: https://www.semanticscholar.org/paper/6f217d984f36499d88ab8a3d89572171552e6f3f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “Prompts and chosen, rejected pairs are selected from MT Bench examples with similar ratings and adversarial data specifically for fooling LLM-as-a-judge tools from LLMBar’s evaluation set (Zeng et al., 2023) (reformatted for RMs).”
ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, methodology): “Considering the challenges in model-based evaluations due to their biases [Zeng et al., 2023], e.g. , open-sourced LLMs climb up on Alpaca-Eval [Li et al., 2023a] leaderboard by utilizing the length bias of the judge GPT-4 [Liu, 2024], we first establish a more reliable evaluation protocol for…”
Humans or LLMs as the Judge? A Study on Judgement Bias (2024, unknown): “3 On the Biases of the Judge”
Disentangling Length from Quality in Direct Preference Optimization (2024, background): “This effect is even observed in in strong propriety models, such as GPT4 (John Schulman et al., 2022), which is now frequently used to evaluate the performance of other LLMs (Dubois et al., 2024; Zheng et al., 2023; Zeng et al., 2023).”