LiveBench: A Challenging, Contamination-Free LLM Benchmark

By Colin White, Samuel Dooley, ∗. ManleyRoberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu and 8 more. 2024

317 citations recorded by the source corpus.

Subfields: Information Systems, Large Language Models, Evaluation & Benchmarks

Citations in the corpus

• From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, neutral): “1 provides a cost-effective and informative evaluation benchmark, and the BenchBuilder pipeline serves as a valuable tool for developers seeking to automatically generate high-quality benchmarks from vast data sources.”
• Kimi K2: Open Agentic Intelligence (2025, unknown): “Finally, general capabilities are assessed using MMLU [24], MMLU-Redux [18], MMLU-Pro [77], IFEval [91], Multi-Challenge [65], SimpleQA [79], and LiveBench [81] (as of 2024-11-25).”
• Gemma 3 Technical Report (2025, unknown): “…LLM reasoning benchmark that aggregates several reasoning tasks (Fatemi et al., Hessel et al., 2022; Kazemi et al., 2023, 2024b; Kıcıman et al., 2023; Nie et al., 2024; Sánchez et al., 2024; Shah et al., 2024; Tyen et al., 2023; White et al., 2024; Yamada et al., 2023; Zhang et al., 2024).”
• Qwen2.5-Omni Technical Report (2025, unknown): “Specifically, we utilize MMLU-Pro (Wang et al., 2024f), MMLU-redux (Gema et al., 2024) and Livebench0803 (White et al., 2024) for general evaluation, GPQA (Rein et al., 2023), GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) for mathematics & science, HumanEval (Chen et al., 2021), MBPP”
• Qwen3-VL Technical Report (2025, unknown): “…et al., 2024), GPQA (Rein et al., 2023), SuperGPQA (Team, 2025), (2) Reasoning: AIME-25 (AIME, 2025), HMMT-25 (HMMT, 2025), LiveBench (2024-11-25) (White et al., 2024), (3) Code: LiveCodeBench v6 (Jain et al., 2024), CFEval, OJBench (Wang et al., 2025c), (4) Alignment Tasks: IFEval (Zhou et al.,…”
• Harnessing the Wisdom of LLM Crowds through Complementarity-Driven Iterative Collaboration (2026, unknown): “However, on VisEval—a task requiring natural-language-to-visualization code generation (NL2VIS), a task type not directly represented in the LiveBench cold-start categories—all routing methods collapse.”
• STMA: A Spatio-Temporal Memory Agent for Long-Horizon Embodied Task Planning (2025, unknown): “5-72b-instruct demonstrates performance comparable to GPT-4o in several benchmark tasks [51].”