Don't Make Your LLM an Evaluation Benchmark Cheater

By Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-rong Wen, Jiawei Han. 2023

247 citations recorded by the source corpus.

Large language models~(LLMs) have greatly advanced the frontiers of artificial intelligence, attaining remarkable improvement in model capacity. To assess the model performance, a typical approach is to construct evaluation benchmarks for measuring the ability level of LLMs in different aspects. Despite that a number of high-quality benchmarks have been released, the concerns about the appropriate use of these benchmarks and the fair comparison of different models are increasingly growing. Considering these concerns, in this paper, we discuss the potential risk and impact of inappropriately using evaluation benchmarks and misleadingly interpreting the evaluation results. Specially, we focus on a special issue that would lead to inappropriate evaluation, \ie \emph{benchmark leakage}, referring that the data related to evaluation sets is occasionally used for model training. This phenomenon now becomes more common since pre-training data is often prepared ahead of model test. We conduct extensive experiments to study the effect of benchmark leverage, and find that it can dramatically boost the evaluation results, which would finally lead to an unreliable assessment of model performance. To improve the use of existing evaluation benchmarks, we finally present several guidelines for both LLM developers and benchmark maintainers. We hope this work can draw attention to appropriate training and evaluation of LLMs.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “Some detection methods have also been built to avoid these cases (Shi et al., 2023; Zhou et al., 2023).”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “This discrepancy arises because benchmark questions typically require responses that are limited to a single option, choice, or word—diverging from the more broad and realistic use cases of MLLMs. Similar phenomena have been discussed in other LLM studies [106, 142, 146].”
Competition-Level Problems are Effective LLM Evaluators (2023, background): “Considering the previous work (Yang et al., 2023; Zhou et al., 2023), although GPT-4 may perform particularly well on some previously seen problems due to its powerful capacity, it cannot be well generalized on un-seen programming problems, and its performance is not significantly different from…”