From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

By Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica. 2024

481 citations recorded by the source corpus.

The rapid evolution of Large Language Models (LLMs) has outpaced the development of model evaluation, highlighting the need for continuous curation of new, challenging benchmarks. However, manual curation of high-quality, human-aligned benchmarks is expensive and time-consuming. To address this, we introduce BenchBuilder, an automated pipeline that leverages LLMs to curate high-quality, open-ended prompts from large, crowd-sourced datasets, enabling continuous benchmark updates without human in the loop. We apply BenchBuilder to datasets such as Chatbot Arena and WildChat-1M, extracting challenging prompts and utilizing LLM-as-a-Judge for automatic model evaluation. To validate benchmark quality, we propose new metrics to measure a benchmark's alignment with human preferences and ability to separate models. We release Arena-Hard-Auto, a benchmark consisting 500 challenging prompts curated by BenchBuilder. Arena-Hard-Auto provides 3x higher separation of model performances compared to MT-Bench and achieves 98.6% correlation with human preference rankings, all at a cost of $20. Our work sets a new framework for the scalable curation of automated benchmarks from extensive data.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

Citations in the corpus

• From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, unknown): “Alignment tuning (Wei et al., 2022a; Ouyang et al., 2022) is a vital technique to align LLMs with human preferences and values.”
• Qwen3-VL Technical Report (2025, unknown): “…Code: LiveCodeBench v6 (Jain et al., 2024), CFEval, OJBench (Wang et al., 2025c), (4) Alignment Tasks: IFEval (Zhou et al., 2023), Arena-Hard v2 (Li et al., 2024d) 1 , Creative Writing v3 (Paech, 2023) 2 , WritingBench (Wu et al., 2025b), (5) Agent: BFCL-v3 (Patil et al., 2024), TAU2-Retail,…”
• Kimi K2: Open Agentic Intelligence (2025, unknown): “0 benchmark, which use LLM-as-a-judge protocols to assess generation quality across diverse, open-ended prompts [43].”
• S-SPPO: Semantic-Calibrated Self-Play Preference Optimization (2026, unknown): “Second, we benchmark against SPPO (Wu et al., 2025), which serves as our primary baseline.”
• Binary Classifier Optimization for Large Language Model Alignment (2024, unknown): “Additionally, it is encouraging that, in the Arena-Hard benchmark, BCO demonstrates superior performance despite having a generated token length similar to that of DPO.”